Overview
Q: Please give us a short summary of your project, how did you get involved and what are the major findings.
A: Okay. It’s basically a mode-effects experiment, looking to see the differences between a computer verses a paper and pencil instrument in the collection of sensitive information in adolescents.
Q: What does mode-effects mean?
A: It’s looking at different methods of collecting data. There are a number of different methods of collecting data; the qualitative types, like ethnography and the like, but the more quantitative are more focused on like, a telephone survey, mail survey, face to face surveys. And as we are looking at, under the rubric of self-administration, different types of collecting information directly from the respondent, him or herself.
Q: What were the key findings of the study?
A: Well, counter to what was believed to be the case in the literature, although the area of mode effects and computerized collection of sensitive information from adolescents is understudied in the literature, the literature that was out there really seemed to indicate that kids were more forthcoming in the computerized administration because it connotes an asocial type of data collection; you remove the person from situation, they entered information right into the computer directly, and so, therefore, the likelihood that others would review it was lowered. Basically, we found in our study, in the mode-effects experiment that we did is that in a school setting, we found the results opposite to that, where some of the literature I referred to just now was conducted in household settings. Well, we did it in a school setting and we found that actually fewer reports of sensitive information were obtained by the computer, visa vie the paper and pencil version.
Theory
Q: What theories helped you formulate your central research questions?
A: Well, it’s not so much theoretical, it’s two outstanding and alternative hypotheses. One is that kids will, like I said, be more forthcoming to the computer because it removes the individual data collector from the situation and once the information is in the computer it will be secure. So, one hypothesis is that kids will be more forthcoming to the computer. An alternative hypothesis is that kids will be less forthcoming in the computerized condition because now, data will be entered right into the computer and therefore it can be whisked away automatically to government entities and accrued in their own personal [ ], I think of a kind of an "X-Files" sort of mentality, where Big Brother has all the information on you and so, the alternative hypothesis is that kids would provide fewer reports of sensitive information in the computerized version. So, it’s testing out directly those two competing hypotheses.
Q: Please explain your research design. How did you decide to pursue your study in this way?
A: The main design questions were focused on the actual user interface of the computerized survey, there wasn’t a lot of literature to guide us, so that was a year-long process. Two of the main areas that we concentrated on were mode equivalence, in terms of the stimulus that was presented to the kid. In some of the previous mode-effects experiments, not only the computerized, but like mail versus phone, it was hard to disaggregate the effects of presentational style from the actual mode. What I mean by that is in the computerized version, you kind of have question by question stimulus, whereas in the paper and pencil, you have simultaneous presentation of the information, so you really couldn’t tell whether it was the method by which the stimulus was presented versus an actual mode difference. Also, the visual interface, you know, how the paper and pencil had a number of questions per page, again, in the computerized version, we had one per screen. So, a lot of the issues around the actual computerized form, where they can move back and forth on the paper and pencil by flipping pages, we wanted to retain that ability in the paper and pencil version, where they could quit at any time, in the paper and pencil, if they were fed up or tired they could just close the booklet, we wanted some sort of computer analog to that, where they could hit a quit button and end at any time. So, one of the main design features, design challenges, was what the computerized form would look like, and we’re constrained because the paper and pencil version had been already designed and printed and that’s something generally that presented a problem and this was not just addressing an outstanding academic question or scientific question, it was trying to address a policy question. The Department of Children, Families, and Learning, which is Minnesota’s education department, wanted to move their students’ survey, the survey responsible for providing a lot of the [ ] prevalence information and the like, profiles of kids generally, wanted to try out a new form of collecting the data; away from paper and pencil to computerized forming. We’re addressing in this experiment, specifically that question, to assess the viability or feasibility of making the transition from paper and pencil to the computer. With that, because we kind of latched ourselves onto this bigger policy issue, we were constrained somewhat in terms of having to adhere to an existing survey instrument that had been administered for several years and limitations on sort of the contextual information that would help flesh, or shed some light on why there would be differences, things like collecting information on distance between the kids. We really had to argue strongly to add the types of contextual information we were able to collect, because it wasn’t a primary analytical focus.
Design
Q: Why did you decide to design your sample in this way; its size and its structure?
A: Ideally, to really test these out, the feasibility, the more policy-oriented focus, and the scientific knowledge or contribution, we wanted to generalize to adolescents generally, which means you want representation of all different types of kids, but we were constrained both in terms of budget and timeline to focus on just a few sites and in the previous literature, some of the mode-effects experiments that looked differences in prevalence rates were compromised, or the statistical power of their estimates, were compromised by the infrequent use rates that were obtained, so really you couldn’t tell the differences between the modes because some of the substances were so infrequently used, some of the behaviors that we’re measuring, such as deviant behavior, were so infrequently reported that the statistical power associated with the comparisons was compromised. So, we decided to focus on alternative education standards, where the kids just by the nature of their admission into these centers, were more likely to have higher use rates, therefore we’d have to collect less sample to attain the same level of statistical power. And we also focused on sites that had existing computer laboratories, consisting of computers that were comparable to the software that we designed. That’s basically one of the big problems we had to address, is where do we test this out. Along those lines, we had to decided whether to do a between school mode-effects, where one school was administered the computerized version and other one was paper and pencil, or within school randomization. And, we went back and forth on that and what we decided is that because of the clustering effect, potentially, within a school, attitudinally, behaviorally, that would have compromised, because of the design affect, our sample size and our sample constitution, we decided to do a within site randomization, which presented its own operational challenges.
Q: So within a school, you would select a class and then randomly assign the students?
A: Right. We would go in a site long enough so that every kid had an opportunity to be assigned. So we’d be there for up to five days and we’d select a site in a time frame where, theoretically, every kid would come to this class, we’d go in there and kind of do the "one-two, one-two" and kids in the paper and pencil version stay and grab the rest of the kids and bring them into the computer lab. So, basically, the [ ] in this particular study, was all kids within those four sites, that were present on the day of the administration.
Q: Could you explain what the students who were assigned to the computer lab did? What was their experience like? Did they use laptop computers?
A: The four sites were selected, the sites varied somewhat in the physical plant they offered. Some sites, the computer lab was basically scattered all over, like within three different rooms, there was primarily PC, but there would be like a PC per table, so there maybe up to like, 15 feet between the kids. Other computer laboratories were very much like what you think of when you think of computer labs in high schools; they’re just PCs, one next to another, and the kids would go in there, we would try to spread them out as much as possible, but sometimes the classes were so large relative to the available computers there’d be a kid at every station, when we had the opportunity, we’d spread them out by a couple. So, we’d give them some general instructions and a computer administrator would start the computerized version and take off for awhile and the kid would go through a tutorial if he or she saw it warranted on how to get through the instrument and how to use the help buttons and the like, and then just take the survey and in the end let the administrator know, he and she would go and answer some contextual information, like distance between the kids, how many kids were there, and then log the kid out and save the data onto disk.
Q: What did the students think would happen to the data?
A: Good question. That’s one of the questions we asked of the kids right before they logged out of the computer. We asked them the contextual information that I referred to earlier, we asked them how private was the situation, we operationalized that to be "How likely is it that others saw you reporting your answers?", meaning people in the immediate surroundings, into "How likely do you think it is that others will look at your data?". And it really varied as a function of the proximity of the other kids, kids that were in a situation where there were a lot of kids really close by, you know they had this computer monitor, 14 inch computer monitor sticking out there, those kids were more likely to indicate that they thought others saw them putting the answers in, including the site administrator, who would walk in, log kids out as needed, would come in and… But generally in our pretesting, as well as this actual experience, the kids seemed to indicate that their data were going to be pretty well protected, but it really did vary as a function of the physical proximity.
Measurement
Q: What was the age range of the students?
A: Twelve to seventeen.
Q: Could you give us some examples of the questions that were used?
A: They’re broken down into three main categories: they’re illicit drug use, including alcohol for these kids, comparable to the monitoring of [ ] items, so we asked about alcohol use, illicit drugs use, such as LSD, amphetamines, marijuana, cocaine, we also asked about… The second broad rubric is antisocial behavior, outside of illicit drug use, which is "Have you damaged property, stole anything, have you been in physical fights, are you a member of a gang?". The third component was antisocial behavior of the family, such as "Does your family have a drug abuse problem, do they have a drinking problem, have they physically or sexually abused you?". So all different levels of what the literature points to as being sensitive types of information including like, suicidal ideation on the part of the kid. So, just a range of different types of sensitive information.
Q: After you collected the data, what did you do with it? Was it coded? Was it coded by someone else? How did you use the data?
A: Well, one of the nice administrative efficiencies gained by the computerized version is automatic data entry, the kid entered the data, it’s in a nice data set. The aggregation of data was probably the biggest challenge associated with the computerized version because we had each individual kid’s information saved to a disk, we had to aggregate all those within sites and across sites, but those data were entered basically, as a result of the computerized version. We had to have the paper and pencil version data entered, there were optical scanning instruments, so that those data, those sheets, were scanned into a data base as well as the administrative questions. We did that under a sub-contract with an optical scanning firm. We got that data set back and merged them and then we had an analysis file.
Q: How were the data analyzed?
A: Well, first we wanted to assess, I keep saying "we" because I did this for our department and also for the Department of Children, Families, and Learning, we wanted to see how well our randomization process was, how well it worked, because it wasn’t the most scientifically rigorous, it was like, duck, duck, gray duck, it was one-two, one-two, come with us. And we knew just because we were out there collecting these data that some kids wanted to be with their buddy, so they knew that we were doing this and they arranged themselves to be with their buddy. And so we had suspicions going into it, around how rigorous the actual randomization process was, so we wanted to see if the two conditions varied at all demographically, by age, race, and the like. And we saw that the randomization process was largely successful, there weren’t any [ ] differences across the conditions and the constitution of the people, of the kids in the conditions. Then we just wanted to test the main hypothesis. Were reports of some of the sensitive information different across the modes? And we were surprised to find out that the reports were higher in the computerized version and we were perplexed, because it did run counter to the [ ] literature, and so we used some of the contextual information that we were able to collect to try to flesh out why that was the case. What figured prominently in those analyses, was distance between kids. We had the administrator enter in how many kids and what’s the average distance and we saw thatæand we did this in multi-variant form, we had dummy variables for distance and computerized version—that when the kids were at least five feet apart, that they’re responses were statistically not different than the paper and pencil. So that’s where you started to see a little more equivalence, is when there was a proximity difference. And we also wanted to see, because the sites were chosen for willingness to participate, existing of a particular type of computer lab, if the differences varied across site, there were some larger sites, although we tried to get a mix of rural, urban, and some sort of representation of minorities. There was one very large site that we hope didn’t drive a lot of the results, and so we added a dummy variable for site, too, to see if there were any interactive effects and there wasn’t, so the results that we observed were the reports of sensitive information were very much, in the computerized version, were very much like the paper and pencil version, when there was a distance of at least five feet between the kids.
Data Analysis
Q: How was the dummy variable used in this situation? Were the students in one school assigned one value and the students in the other another value?
A: We had a dummy variable for each site, well k-1, so we had three dummy variables for the four sites. And we didn’t see any interaction between those dummy variables in the results. I should say that, in the first analysis, when we were looking at site differences, or condition differences across reports, it was basically [ ] because we wanted to look at each type of report. Kind of binary coded, "did it occur, did it not occur?". In the multi-variant analysis, we had the counts of the number of sensitive reports provided, so it was a count variable, so it was a continuous variable and therefore we used ordinarily squares regression.
Q: Did you use statistical significance tests to draw inferences from the data?
A: Yes, on all those. For each of those three levels testing of the methodological integrity, the condition differences, and the effects of contextual and site on results that we had obtained earlier, in the bi-variant form.
Q: Do your findings differ from those that are drawn from household surveys of adolescents?
A: I think that’s one of the biggest policy implications, policy and scientific contributions, is that there have been pushes at the national level to computerize the Monitoring the Future survey, which is a student survey, they’ve already computerized the national household survey of drug abuse, which is a face to face household survey, where the kids will enter the information into a computer. What this shows, is that if the Monitoring the Future wants to adopt a computer methodology, they have to the proximity variable into account and try to, as much as possible, provide private situation, or a private area for these kids to enter the data, otherwise, you will likely observe the results we did, where it just runs counter to the results that have been obtained in primarily household surveys, addressing this question. That warning, or that admonition, is kind of…there’s a big caveat, because we had some artifacts associated with this particular study, that I don’t think the Monitoring the Future would. The whole study, the data were collected by state-sponsored researchers, in house, and so, for example, in an alternative education center where a lot of kids were used to their teachers and the like, here we come in, with our ties, we look bureaucratic, we may have enhanced the Big Brother perception, we’re coming in there, it’s very artificial, they don’t know us. Also, having the state-sponsored researcher go around and log kids in and out of the survey may also have enhanced, or exacerbated the feelings of Big Brother. So, I think that that’s a limitation of this study and that perhaps this Monitoring the Future study relies largely on existing staff to orchestrate the collection of those data and so I don’t think that they’ll have that limitation, but it’s something to be…before this transition to the computerized data collection is made, more research investigating the effects of proximity, state-sponsorship of the researchers, and maybe a replication in a public school sample versus an alternative education center, maybe warranted before the transition is made, otherwise it will artifactually affect the prevalence rates obtained.
Q: Are you doing that kind of study now?
A: We’re waiting for the appropriate [ ] to come out, to write the proposal. We’re kind of chomping at the bit to move to the next step because it was unexpected. Naïve Tim, in his dissertation, had all the hypotheses [ ], you know, I’d expected the prevalence rates or the reports to be higher in the computerized version because relying on [ ] literature and I was disappointed first, because we found just the opposite, but really it’s interesting in that you really can’t translate from one set of studies to all applications and it really goes to construct validity, where construct validity is based on a series of investigations done with different populations, done in different situations, so the robustness of the findings obtained earlier on was compromised by their focus on a particular context, or environment. So it’s excited to think about how to manipulate these variables that may be accounting for the unobserved, the unexpected results.
Q: What other implications did the results have?
A: Regarding the policy application, we did it for the Department of Education, those plans were stalled, they wanted to do it this next round, in 1998 we administered, with the Department of Education, the Minnesota Students Survey, this school survey. They wanted to have at least half of them done in computerized form. That’s kind of stalled that, before that we tried to tease out exactly what sort of physical plant is needed to get the best data possible, so there was a slowing down of this rush towards computerization.
Interpretation
Q: Did you find some interesting skip patterns in the data?
A: We really did, where in the computerized version, and again, I think it’s a result of the presentational styles of the two different modes; the paper and pencil we had this skip instruction in there, couched in this whole page of maybe ten other questions, if you haven’t used alcohol or drugs, please skip past all this specific drug questions. And in the computerized version, it was a serial presentation, so each question came up, including the skip instruction, if you haven’t used alcohol or drugs please…we actually asked a question, "Did you use alcohol or drugs?", if no, then it skipped them automatically to the end of the questionnaire, and the instruction actually said that if you select this, you will skip, these more detailed instructions, and I think kind of a combination of the kids saying "Hey I want to get done with this thing quickly", more kids in the computerized version followed that skip sequence appropriately than the kids in the paper and pencil version. I think it’s a result of the obviousness of the instruction and the automatic skip, because we looked at, within the paper and pencil version specifically, the kids who had said no they don’t use drugs, a lot of them went through and answered the questions anyway and the specific questions indicated that they did use some illicit drugs. So, we’ve kind of forced them, in the computerized version, to automatically, if they select it they’re automatically skipped, but the implication of that is that when you have a skip sequence, which is one of the [ ] aspects of a computerized self-administered data collection instrument is that you can have all these embedded skips and it’s more likely the kids will follow it, that they may be skipping for reasons other than the intended purpose, they might want to streamline it, or maybe [ ] to provide such information, so it’s a warning to consider what the user interface looks like
Q: How would you avoid this problem in the future?
A: I think malleability on both sides, on both modes, we were, like I said, earlier, were constrained to be, in terms of the user interface and the content, to be exactly like this existing instrument, in terms of its formatting, its content, and so we really had to force the computerized version to look like the paper and pencil version. We’re doing a study now where we have the ability to look to make both forms look equivalent, so the computerized version is going to have a full screen of questions that looks very much like the paper and pencil version.
Q: You've mentioned the importance of policy makers as an audience for this research. Are there other audiences for the research as well?
A: The results of these studies have resulted in three journal, two published, one in press, so they really contributed to the understanding of this issue, in the scientific literature, I think they’re surprising to many of the researchers who have said the evidence clearly indicates that kids will be more forthcoming in the computerized data collection, and so I think it’s a contribution to the understanding and the scientific literature as well, but for the purposes of this study, and the purposes of the funding, it was funded by the Substance Abuse and Mental Health Services Administration, who want applied, policy-relevant research, the primary interest was assessing the feasibility of application in our study survey. The secondary was a contribution to the scientific literature and I think it accomplished both.
Q:Most graduate students in the social sciences are headed towards careers that combine teaching and research. You've been engaged in this enterprise on a full-time basis now. Do you have any thoughts about the tradeoffs between full-time and part-time research careers?
A: I, by design, throughout my graduate career and undergrad, wanted to work in an applied setting in social research because I wanted to see the transition from the research that I was involved in to policy more closely linked, so a lot of the questions I address in my research have been generated by from a policy perspective, generated by the legislature, generated by Congress at the federal level, and so it operationalizing those larger policy relevant questions into a research study; undertaking this research study and then turn it around to summarize, not in terms of just what the data say, but what it means in terms of policy. So, throughout graduate school, I worked full-time in applied settings trying to get a sense of how research contributes to an applied problem, a policy problem, and I find that very rewarding in its context. It does have its downside, is that sometimes your choices are politicized, you have to attend to a number of different constituencies when you’re framing your hypothesis or selecting a particular methodology, and just because I’m working in a policy research area doesn’t mean that my results are going to be used by the policy makers. I still have that bridge where…I guess, the difference between the way I saw it was, in academic research, and I think this had been embedded in academic research in sociology for quite some time is that it’s not part of the sociological perspective to do research that is applied directly. Research is undertaken and then they can be used by applied people as they see fit, it’s not "do it specifically for policy purposes", because they don’t want to sully the research process, the scientific process, by taking the political into account. I accept that, in my new position, I want to be sullied, I want to take into account the political needs of this research and when I write the reports I have two different versions, one for public consumption, one is for legislatures which means bullet points, not talking about using multi-variant analyses, or problems with design effects, and the things that we’ve covered in this discussion of the methodology associated with the study, but summarizing the results and what does it mean to them, so it’s a different type of writing. In an academic type of writing, you focus on the methodology, same with the presentations, you focus on the methodology used, on the appropriateness used in the statistical tools, statistical methods applied, and then let the results stop at just interpretation of the results, not going on to policy implications. I’m pretty satisfied with the situation I’m in because I think I can do really good research that contributes to a policy question as well as is the case in this study, contributes to the scientific knowledge.
Q: How would someone pursue a career like this?
A: We oftentimes have internships available, government agencies oftentimes have internships, where they can work on a policy question and help in the data collection, data analysis components of that. But largely, a new graduate student, do what I did and what kind of by necessity a lot of students have to deal with when they come in is, hook up with a prof. or a researcher on campus and get some experience. Cleaning data, pulling together journal articles, just moving away just from the book stuff to the actual real-world stuff. I remember when I took stats as an undergrad, things always worked out perfectly when I analyzed the data, the data were sanitized for me, and I remember when I took my first job out of school, I looked at the data and I said "no fair", I mean, I don’t understand these data, you know, there are too many missings, I don’t understand that. And so, more real world experience, working on applied research or research with an investigator, I say, by necessity, because oftentimes that’s how people get research assistantships, and so, I get a lot of residents and CVs from recent grads and there’re two different types: one, that has, they’ve done very well in graduate school, 4.0, got out in four years, God Bless ‘em, you know they did a very nice job from an academic perspective, but really, there’s no real world experience that I can see, having to deal with long hours, working with people, productivity, there’s no way to assess that. The other type is, "I worked as a graduate RA throughout", that shows me that they’ve been able to juggle multiple needs, here are some of the reports they’ve worked on that demonstrates that they can synthesize what they’ve worked on into a form that consumed by the public, so something to demonstrate the skills as they would apply to the job to which they are applying. There are a lot of very smart graduates of a master’s program or a doctorate program, or even undergrad, that are repellent as a perspective employee because they have no sense of how to apply what they’ve learned, so a demonstration of what they’re learning in their coursework is almost a necessity if they want to make the transition to a type of…if they want to be like Tim.