New Survey: Mormon Culture and Belief
Ok, so here is our new survey, which focuses on Mormon culture and beliefs. The triple focus of this survey is to measure faith commitment, church engagement, as well as cultural belief on a variety of topics. The survey is 60 questions in length, and so may take as long as 6 minutes to complete. Some of the questions, due to their connection with obscure doctrine, may seem offensive to you. If this is the case, we apologize in advance.
Our goal is to get 5000 responses to this survey, and we will attempt to share this survey via Facebook, this blog, and email invitations. As soon as we feel the responses are no longer coming in, we will prepare our findings and publish them.
To invite friends to take the survey, just pass them this link via whichever method you prefer:
http://www.surveygizmo.com/s/271583/lds-survey
Thanks for your participation and support.
I believe that in a few cases, it would have been helpful to have a fifth choice: namely, “I do not understand the question.”
In such cases, I answered, “neutral”.
Comment by Matthew Chapman — April 4, 2010 @ 1:10 pm
Seemed way too long to me.
You want 5000 responses? Wow.
Comment by Geoff J — April 4, 2010 @ 1:40 pm
Seemed way too short to me. You need to add 40 questions to round out at an even 100.
Comment by Jacob J — April 4, 2010 @ 1:54 pm
Matthew: That is why neutral is an option.
Geoff: It was a little long. But I wish I could have added more questions. It was a tough balance to get to.
My wife wants 5000 responses. That gets our confidence interval to 1 in comparison to the assumed population of the church. I’d settle for a confidence interval of 3 (1000 responses).
Jacob: I wish I’d asked so many questions I didn’t think of until it was too late. There is always the next survey, I guess.
Comment by Matt W. — April 4, 2010 @ 2:23 pm
That gets our confidence interval to 1 in comparison to the assumed population of the church.
Um, I hate to break it to ya, guys, but that only works if you have a random sample. Good luck getting a random sample of Church members via the bloggernacle.
Comment by Last Lemming — April 4, 2010 @ 3:43 pm
On questions that were worded in a way that I felt that I could not answer properly, I checked neutral.
Comment by Bob — April 4, 2010 @ 3:51 pm
LL- we are primarily relying not on the bloggernaccle, but on viral mormonism via email and facebook. I’ll try and qualify all that in my conclusions.
Comment by Matt W. — April 4, 2010 @ 4:31 pm
It will be interesting to see the results!
Comment by E — April 4, 2010 @ 5:02 pm
Very cool stuff; I can’t wait to see the results. I’ve sent the link to Facebook, so count on at least 2 more people taking the survey (yeah, I’m that popular).
I had a very hard time committing to an answer, even with the neutral option. There was a strong urge to qualify every decision with an explanation of why. I particularly agonized over this question:
39. Christ suffers because it hurts him to be in a relationship with sinful creatures.
But that is the nature of surveys, so what can you do?
Comment by Tristin — April 4, 2010 @ 7:11 pm
Done. Will let others know, too.
Comment by m&m — April 5, 2010 @ 1:11 am
Thanks everyone. And thanks for passing it on. We are at 200 responses so far, so a long way to go…
Comment by Matt W. — April 5, 2010 @ 8:00 am
This was fun. I’m excited to see the results.
Comment by Todd Decker — April 5, 2010 @ 9:33 am
First, great survey. Second, do you happen to know if there is a common type of person who visits and participates in the Bloggernacle and/or NCT? What I mean is, if the typical blogger, for example, is unwilling to be anything but neutral on questions of doctrine, feels out of place in their ward or the Church, and has generally unorthodox views, then most of those who encounter and take the survey would mostly be representative of this type of person, no? And wouldn’t that ultimately bias the survey in a particular way? Is it possible to be sure that you are getting a fair representation of the wide spectrum of Mormons commenting on Mormon culture and belief?
Comment by Jacob B. — April 5, 2010 @ 10:36 am
Oops, just noticed that my question was already somewhat addressed in earlier comments. Sorry for the repetition.
Comment by Jacob B. — April 5, 2010 @ 10:38 am
Heh. Not having been over here in awhile, I just found this survey in a link from a facebook friend. Don’t normally do surveys, but did this one just for fun. It was a usual survey at first, but then all questions on spirit birth, soteriology, and Heavenly Mother! I’m thinking to myself, “I wonder if the guys over at NCT have anything to do with this?”
Comment by Eric Russell — April 5, 2010 @ 4:53 pm
Ha! Wow. That is pretty funny Eric. It also proves that I have underestimated the reach and speed of some of this social media when it comes to things like this…
Comment by Geoff J — April 5, 2010 @ 5:19 pm
We’ve got 296 responses in 30 hours. Not bad for viral marketing. Thanks to everyone who is spreading the word.
Comment by Matt W. — April 5, 2010 @ 7:20 pm
I think this is really interesting. Good Luck getting the numbers that you want. I have sent it to the British people that I know.
I would like to have seen a nationality/location question. Only because I wonder whether there is increased diversity in some areas as opposed to others.
However, I think you have done a good job considering the difficulty of the topic.
Comment by Aaron R. — April 6, 2010 @ 4:10 am
Aaron: I get Location for free via IP lookup, so didn’t use a question for it. So far we have responses from the UK, Norway, Italy, Bahrain, Israel, Guatamala, the US, Canada, and a few others…
Comment by Matt W. — April 6, 2010 @ 6:58 am
Sorry. I should not have doubted that you would have had your bases covered. Good Luck.
Comment by Aaron R. — April 6, 2010 @ 7:24 am
No it’s a good question. And sadly, there are some bases I did miss covering (educational attainment for one!)
Comment by Matt W. — April 6, 2010 @ 9:34 am
I used your questions to stimulate discussion at our FHE last night. I have 3 teenage daughters, raised in the Mormon Church and attending Seminary currently. Many good ideas and questions were discussed as a result. I and my wife have both done the survey, with somewhat different answers on several questions. The discussion at FHE also made me realize again that my daughters are not getting much “meat” in their church education.
Comment by kevinr — April 6, 2010 @ 11:30 am
Took it – I thought you did a good job putting this together. I’d love for us to link to your survey from MM if you are interested. I suggested it to one of our admins (who also forwarded it to me in the first place).
Comment by hawkgrrrl — April 6, 2010 @ 1:35 pm
Matt W. I don’t know if you think it would help, but you could post this on MM as a guest if you want. It might drive some extra traffic your way. If you think it would help, email me?
Comment by Aaron R. — April 6, 2010 @ 2:15 pm
Setting aside the issue of randomness, a 99% confidence level is way more than you reasonably need for something like this. Unless you believe that people who answer later are different from people who answer earlier, take your first 1067 and call it good. This gives you a margin of error of 3% at 95% confidence. You’re currently shooting for a maximum of plus or minus 1.8% at 99% confidence. The size of the population of church members is irrelevant for the population proportion sample size math.
Actually, it would be interesting to compare the median answers of your first thousand responders to the second thousand (or later). You might see differences in those who read blogs regularly and those who read them less regularly.
Comment by Owen — April 6, 2010 @ 3:18 pm
I’d love to see a question about the appropriateness of playing cards and Dungeons & Dragons…
Comment by Dane — April 6, 2010 @ 5:16 pm
Aaron and Hawkgrrl: It’d be great to link to at MM. I think you attract a bunch of people we’d otherwise miss. If you want to post it up as a Guest Post, by all means, feel free.
Owen: We are at around 360 respondents now. Basically, I am going to give it the month of April, then kill it so I don’t have to double my cost for having the ability to get 5000 respondents. I’ll be delighted if we hit the 1000.
Dane: too funny, we’ll have to do that on some other go around!
Comment by Matt W. — April 6, 2010 @ 5:51 pm
I found the survey through Mormon Matters.
Sadly, many of the questions on the survey left me without a satisfactory response option. How can you ask “Does Heavenly Mother exist?”, then “Is Heavenly Mother equal to Heavenly Father?”? For people who don’t believe or who are not sure about HM, how can they even answer the second question?
Really, I think the intent of the survey is good, but I don’t think there was appropriate space provided in the response options to accommodate other opinions, especially as it pertains to doctrinal (as opposed to practical) matters. I suspect that the validity of the responses to the survey will be called into question as a result. Also, as has been mentioned in a comment above, requesting survey participants through facebook, blogs, etc. isn’t a way getting a random sample of members of the church, and will prevent you from drawing any statistical conclusions based on the data collected. Finally, if you were planning on publishing this study in an academic journal you would have needed IRB approval to use human subjects in a survey so that data could be published based on human responses, and the human subjects would need to be notified that their participation in the survey was a form of consent to let their responses be published sometime in the future.
Still, I think the concept is good. If there could be a way of creating a random sample of LDS facebook users or LDS bloggernacle participants or some other online community, you might be able to create a survey that sheds light on the opinions of those limited communities of users, but it would still be impossible to extrapolate those opinions to the greater LDS population in general. As has already been pointed out, the typical Bloggernacle participant is probably quite different than the typical member of the LDS church.
The biggest worry I have about surveys like this are what people are going to do with the data, especially if they do not understand the faults in the design of the study. Perhaps many who read the results will rely on them and begin to believe that most people are/are not fulfilled by the music in church services on Sunday, or that 80% of LDS church members read/do not read their scriptures every day, when really nothing of the sort can be confidently stated based on the (future) results of this survey. I suggest that Matt W. and all of his co-investigators should exercise extreme caution in reporting the results of this survey. Ideally, they should start over and try to find a way to do it more correctly from the get-go. I’m sure there are many statisticians in the bloggernacle who could advise.
Comment by SteveS — April 8, 2010 @ 3:14 pm
SteveS:
If you are worried about bias in the sample group, the best way to counter that is to do as the original post asks, and feel free to e-mail this survey to your normal LDS friends. Since the totality of the bloggernacle seems to be about 300 regular people with another 700 or so lurking (best guess), that’s one of the reasons we really wanted to get a larger sample size.
In any case, we will do our base to delineate any potential bias in the survey as we publish our findings. For example, we know our sample group skews male at this point (So get your wife to take the survey.) We also can see that more people have come to the survey via the blogs than via facebook, but most have come via email. We have a very wide spread across the total US, and a pretty solid showing in the UK right now. We also have respondents in many other countries coming in. WE also Skew towards people who are very active in church, which is not surprising. We are just under 500 responses, which is good for 1 week.
Also, the invitation did not that survey responses would be used to “prepare…findings and publish them”, so I think you are being silly.
Anyway, thanks for your thoughts and opinions. Feel free to invite more people to take the survey.
PS- re: the heavenly mother question, we will be correlating the two questions on the back end.
Comment by Matt W. — April 9, 2010 @ 6:40 am
Matt W.: Getting my wife or any woman to take the survey to prevent “male bias” doesn’t do anything to help the data be any more statistically viable. Even if the respondents came out 50/50 male/female in the end, you would still have the insurmountable problem of a lack of randomization.
And of course you ignored the biggest point I was trying to make about the dangers of conducting a flawed study and then publishing the results, which have the potential of influencing the perceptions of people who don’t know how to critically evaluate the design of the study to determine how much credence to lend to the results. I feel it is irresponsible at best and deliberately misleading at worst to plan on sharing the results of _this survey_ to the world. I’m not trying to slam you or the idea of trying to understand LDS attitudes and beliefs better, but I _am_ criticizing your methodology in hopes that you can adapt and reconfigure so that you can produce statistically viable data. Cheers.
Comment by SteveS — April 9, 2010 @ 7:53 am
SteveS- I think that’s a silly assertion. Every study is flawed. If I increase my sample size to the appropriate degree, based on basic concepts, I’m virtually guaranteed natural randomization. I don’t have any means of controlling the randomization. I am merely attempting to get the survey out to as many venues as possibly. I can do a strict analysis and check how well me sample set links to the mainline of the church, and yes, it is true that it currently does not strictly represent the church totally. It mainly represents 25-35 year olds who are active, married, calling holding members of the church in the US, typically leaning toward the tech-savvy category. I can tell that via the people who are willing to answer the survey. From anecdotal criteria I can also see I am losing the far left and far right in the survey for various reasons. I can spell all that out in my results.
The biggest point you’ve attempted to make is that my study is flawed, when in fact I have, as of yet, made no study. I am merely collecting data. Thus I must say your attempt to undermine my study by pointing to flaws that do not yet exist is…well…flawed.
Comment by Matt W. — April 9, 2010 @ 10:08 am
Nevermind, Matt. You’re clearly right. I’m silly and you its totally true that sample size negates the need for randomization. And I should have known that you’re only collecting data and have no intention of studying it. Good luck (getting anyone to take your survey seriously).
Comment by SteveS — April 9, 2010 @ 11:54 am
SteveS: Your mom.
Comment by Matt W. — April 9, 2010 @ 2:30 pm
SteveS: I feel it is irresponsible at best and deliberately misleading at worst to plan on sharing the results of _this survey_ to the world.
Ha! And I feel you are kind of a spaz SteveS.
Good grief man, this is an amateur online survey and doesn’t claim to be anything else. How seriously are you expecting the world to take such a “study”?
Comment by Geoff J — April 9, 2010 @ 3:54 pm
Geoff J,
So I’m confused… should I not be anxiously waiting to blindly base my whole worldview on the outcome of this survey?
Comment by Riley — April 9, 2010 @ 4:33 pm
Riley: mainly just the type of car you buy, of course.
Comment by Matt W. — April 9, 2010 @ 7:25 pm
IN RE SteveS vs. Matt W.: I remember a story of when a bunch of university brainiacs fussed at the writers of the Superman Comics about a trip Superman took to a far off galaxy. They pointed out that Superman would had to have exceeded the speed of light and thus violated Einstein’s theory of relativity.
Said the writers: Einstein has proposed only a theory when Superman is a fact.
Comment by mondo cool — April 9, 2010 @ 9:06 pm
Interesting survey; curious to learn the final results.
I agree with several others about the wording in the survey being misleading for several of the questions. For example, this statement: “The Bible is the actual word of God, to be taken literally.” While I believe the Bible is the word of God revealed through his servants as far as it is translated correctly, I do *not* believe it is to be taken literally. So as others have mentioned, I assumed the one cancels out the other and chose neutral, although I don’t think that really serves any purpose in helping you clarify what I (or any other respondent) believe about the Bible.
Also, initially, numbers 25 and 33 seemed redundant, but I then realized the difference in creative attribution. I know nothing about survey writing, but wouldn’t it be more helpful to pair these to avoid confusion? (For reference: #25. God fashioned our spirits from a substance we call intelligence, or from ’spirit element’ and #33. Before God gave us our spiritual bodies, we were eternal beings called intelligences.)
Comment by Jen A. — April 9, 2010 @ 10:20 pm
Mondo- If I ever do this again, maybe I’ll include a question about superman…
Jen- Confusion was intentional.
re: the literal bible question- this question is actually one of three requirements of the definition of an Evangelical Christian, and is taken almost verbatim from their terminology. More on that later.
Comment by Matt W. — April 9, 2010 @ 10:32 pm
Good news — it seems to be going a bit viral on FB, at least among my friends. And most of those are pretty regular Mormons, so that helps normalize the sample.
Comment by Melynna — April 10, 2010 @ 8:35 am
That’s great Melynna, thanks for letting me know.
Comment by Matt W. — April 10, 2010 @ 12:17 pm
I answered “neutral” to all of the Theological Questions because I don’t believe any of that stuff… the origin of souls, the Nature of Heavenly Father … can be definitively understood by human beings.
Comment by V the K — April 10, 2010 @ 7:56 pm
When do you expect to publish the results?
Comment by Clark — April 11, 2010 @ 12:38 pm
I have until the last day of April to get respondents. So sometime in May, I will start writing up different cases from the results set. I have just under 600 responses right now, so I’m hoping I can get more viral spread of the survey and get more responses from non-bloggernacle sources. One Fascinating thing to me is the feeling that because the survey has a subset of respondents from the bloggernaccle, it will skew liberal. I find this odd, since a few years back we did a poll that showed the blogs actually skew 59% republican (Which is not too far from the church’s 69% republican)
Comment by Matt W. — April 11, 2010 @ 1:27 pm
Religiously liberal or politically liberal? There are lots of religiously liberal but politically conservative folks. Of course I like to think I’m conservative in both senses despite believing in evolution. (grin)
Comment by Clark — April 12, 2010 @ 6:39 pm
“One fascinating thing to me is the feeling that because the survey has a subset of respondents from the bloggernaccle, it will skew liberal. I find this odd, since a few years back we did a poll that showed the blogs actually skew 59% republican”
I didn’t see any political questions on the survey. Wouldn’t you say there’s a difference between being religiously liberal/conservative versus politically liberal/conservative?
Comment by Clean Cut — April 15, 2010 @ 6:05 pm
Whoops–I see Clark asked essentially the same thing. I stopped reading after comment #44.
Comment by Clean Cut — April 15, 2010 @ 6:07 pm
Done. Good stuff. I was also hesitant on some of the cosmological questions, mostly because my views are in flux.
Comment by Ben Pratt — April 15, 2010 @ 10:01 pm
Q48 is not understandable; Q49: “all” or “some”?; Q50: by whom – God or members?; Q57:”the”, as a definitive article, means “all” but you don’t mean that. Wow! My silence on the rest must mean you did well. Thank you for your time, energy and creativity. I’m looking forward to the results.
Comment by Nathan Sorenson — April 19, 2010 @ 10:08 pm
Please consider rewriting some of your confusing questions. Neutral was a favorite answer on those. In my opinion, my answers are not 100% accurate due to the confusing wording in some questions. Wording matters immensely when it comes to religious beliefs!
Comment by you you — April 23, 2010 @ 8:25 am
As someone coming from a market research background, I commend you for the effort but it seems the greatest fear I have is how the results will be interpreted. It’s absolutely normal to confront survey participants with questions that force them to make tough choices and also that allow for comparisons between answers. This allows for correlation and identifying variances in stated beliefs vs. actual beliefs. However, I think my sensitivity is that the more obscure doctrines require qualification when an answer is made.
For instance, “There is no contradiction between the theory of evolution and my religious beliefs.” Are we strictly talking about THE theory as defined by Darwin, Mendel and expanded by others which includes statements that man and primates have a common ancestor (that man literally evolved from a primate) or are we talking about evolution as a biological concept and principle of nature?
Or, “People who are excommunicated from the church are doomed to outer darkness?” Depends on how they respond to the excommunication and how they lead their life afterward. Hard to answer that with your Likert scale.
48 – as someone else stated – what? I know exactly what you’re trying to ferret out, but this question could have been worded differently to improve usefulness of the response.
49 – are we saying just baptized or are we including all of the ordinances of salvation? If you meant all of them, then you should have said that.
42 – when, in this lifetime or after Christ returns?
36 – “God lives outside of time” – our time as we who live here on Earth consider it or time in general?
So while you’ll get data out of this on the challenging doctrinal issues, I question the usefulness of it for drawing conclusions other than that many members aren’t familiar with the doctrinal questions at issue and even then you won’t know because many answered neutral because they didn’t know what else to answer.
As for the other questions, they will be useful in understanding how a certain segment of the demographics you’ve attracted feel about their faith, responsibilities and church attendance. But if you wanted useful conclusions other than a bunch of members who are net savvy believe X then you should have included more demographic questions.
Comment by AlainB — April 23, 2010 @ 9:17 am
I understand this is a survey, but I would really like to have given written response to half of the questions. I didn’t respond to a 1/4 of them as I couldn’t tell if you were fishing for an understanding of complex doctrine or if the questions were just poorly written.
Comment by Mike — April 23, 2010 @ 9:22 am
Matt,
Apparently you are getting a lot of non-NCT readers taking the survey because none of them saw your first post asking for help on the wording of the questions.
Comment by Jacob J — April 23, 2010 @ 5:28 pm
AlainB- my biggest regret is not including more demographic questions. Good Call, as it is, I think I have a pretty good basis, and since I am using multiple channels to promote the survey, I hope I am getting past the new savvy only problem. I am attempting to track for that, but we’ll see how it turns out in the results
Nathan, You You, and Mike- yep. Nothing I can do about it now. Once you release a survey, you can’t “rewrite”. It would ruin the survey results.
I will call out in the results where there were concerns with certain responses.
Jacob J- Yeah, this survey has definitely gotten a lot of “non-NCT” attention. We’ve kind of made a fun social media experiment out of it, between the blog, email, twitter, and facebook.
Comment by Matt W. — April 23, 2010 @ 5:44 pm
I think this survey is too badly worded to measure what it intends to measure. I appreciate the effort, though.
Comment by Fish — April 27, 2010 @ 9:48 am
The question regarding Heavenly Mother’s equality was difficult to answer. Does the question mean (as I took it to mean) equal in value, or does it mean equal in function. While one might value Heavenly Mother and be comforted by her existance, that doesn’t mean one might feel it apporopriate to pray to her.
Comment by Michelle — April 27, 2010 @ 11:43 am