Hello everybody welcome to the stats for the masses webinar design research process design research process design research process method.
What’s the difference my name is Elaine Eisenbeis I’m the owner and principal statistician here at Omega statistics.
Thanks to everyone who’s taken the time to join us live today this is actually a favorite web webinar for for our fans it’s a personal favorite of mine too so what we’re going to cover today are some the differences between design research process design research process and design research process method our last webinar and I think it was September was about the literature review so typically after you get your literature review done your next question is well okay what am I going to do how am I going to set this up right so that’s what we’re going to cover today the well-planned study it’s the only kind work that that’s worth doing right working a bit more at the front end of your design research process I’m taking a time to think about the logic as well as the logistics of your study will save you time money it maybe might even add a few years to your life today’s just an overview but I hope that the information you receive today will give you some ideas and guidance on planning your design research process I say this in all of our webinars but use what you learned today apply it in a holistic way as a design research processer it’s important to look at the big picture and it’s important to take into account what can go right and what can go wrong there’s always limitations in every study so you want to take those into considerations too and looking at the big picture will help you to see these things this presentation is probably going to last about 4550 minutes and then we’ll take some time at the end to answer some questions however I’m going to take some time as I go through the presentation to pause and ask for questions from from the audience so you don’t have to wait till the end and and so maybe I’ll get some help with with rod who is here today helping me I’ve currently muted the line so only my voice could be heard but rod is our board operator and he’s a with Silverado Creek studios and we have him to thank for helping to get our recordings in top shape before delivery to our viewers if you call the Omega Statistics offices he’s the voice you hear on hold or if you call after hours he’s got that wonderful voice that local radio station calls him mr. golden honey voice if you stick around for the QA yeah you’ll get to hear what I’m talking about so in the meantime though if you have any questions feel free to type them on the chat board and like I said I’m going to stop every now and then through the presentation and rod will ask me any remaining questions we have the end of the presentation and yeah we do have the same last name it’s rod Eisenbeis ah he’s my husband but he’s awesome he’s great so he’s the best if you have any technical issues I often find that just logging out and back into the webinar clears up just about everything so when in doubt reboot and let’s get started now everybody sees a slide every webinar nothing much has changed I am a private practice statistical consultant I’m based in Southern California I do medical clinical biotech design research process specialize in critical clinical trials all the phases i I do work on methods and I also work on data analysis so I do design research process and analysis I do help with dissertation assistants somebody did type in when they when they signed up for the webinar they wanted to know are you available for hire for PhD students advise and/or contract for design research process related support yes I am that’s probably what we do best here at Omega statistics is helping the dissertation clients it’s one of my favorites and then of course the clinical design research process we also help with survey design research process also with the big data kind of things predictive modeling exploratory design research process things like that.
I work on projects just about of any size from individual design research processers working on your journal articles the dissertation students small businesses and then I I’ve worked with large corporations like Amgen I’ve done the lean thirteen program for Nutrisystem I did the analysis on that so yeah I the nice thing about being a statistician is you can play in everybody’s backyard and that is very true and our motto is to provide elegant solutions effectively applied so we’ll start with what is design research process design research process we’ll start with design research process and as we go through this webinar you can see how it kind of just flows right into the methods so there’s not really a stopping point where you stop a design research process and you go into methods they are very much related however you kind of need to think about the design research process first I think I already mentioned that design research process is Liz logic methods is the logistics so the design research process is you know the framework you know basically what you’re building how you’re design research processing it as far as the theory and the logic of it and the methods are your nuts and bolts your tools that you’re going to use to actually get the answers that you’re looking for so the textbook I always like to show the textbook and then kind of make it a little bit simpler the textbook definition a design research process design research process is it refers to the overall strategy that you choose to integrate the different components of the study in a coherent and logical way they are by ensuring you will effectively address the design research process problem the stats for the masses definition think of design research process as your recipe okay what are you making how look you know are you making a cake are you making a souffle are you making a you know a roast out you know what are you making what ingredients do you need how will you cook and serve it will it be tasty you know is it going to be good you can also think of design research process design research process as a blueprint for collecting measuring and analyzing the data it’s your blueprint so a recipe blueprint the design research process problem is going to determine a type of design research process you will use not the other way around so you do your literature review and you have a good idea now of what your gap is what the problem is you want to address and then you’re going to design research process your study around the problem the best design research processs best study design research processs have repeatability which means that they’re laid out in a way that another design research processer can read your port look at your study and they can reproduce it okay they can they can run it and see if they can get the same kind of results so you want to have enough information and detail it enough so that somebody else can come along and try to reproduce your results also generalizability not only will the results apply to the population you sampled for your study but when your study design research process is repeated with that other another sample from the same population the results should be similar so you want to try to put that into your design research process usually a generalizability is going to be mostly with an experimental design research process you’re not going to see it very much with other design research processs and we’ll cover it a little bit more as we go through things.
I’ll talk about a little bit more later when we get to the experimental design research process part but the repeatability or reproducibility and generalizability give more validity and impact to the results of the study we usually design research process a study for the repeatability and generalizability it’s a bit harder to come by like I said it only happens by repeating to study over and over if you don’t have an experimental design research process basically only if it’s a correlational study something like that you’re not going to get generalizability the only way you’re going to get close to it is to have the the study done over and over and over and over again by different people with different samples basically now I’m it’s an overview today ok so there’s probably many many design research processs I’m not going to cover I’m not even going to mention but there are basically three types of design research process design research processs there’s descriptive design research process and it’s basically what it is it describes it describes the subject or at a phenomena with words numbers both sometimes it’ll it’s as simple as you know here’s a means and standard deviations and here’s the counts of how many people we had in each group and you know maybe there was some observation done and it’s basically just detailing out what you saw descriptive studies you’ll mostly see those in a in a qualitative type of study and we’ll talk about that in a little bit but you know it’s basically just a description not really statistically involved at all just descriptions there’s correlational and a correlational study measures the relationship or association between variables it’s basically we’re going to have one we’re working on today where the Sun rises and a rooster crows what’s the relationship how are they associated it’s what’s the strength of the association there then there’s experimental and like I was saying that’s typically the gold standard that’s if you can do this this is what you want to do but most studies especially if you’re working on a dissertation if you’re into social sciences if you’re in some of the soft sciences you’re not going to really have an experimental you’ll probably have more of a correlational maybe even descriptives but experimental is when the design research processer will manipulate the subjects to investigate a cause-and-effect relationship so if you do something to one group but not another group does it change the outcome so a lot of times in clinical design research process you’ll have a treatment and control group and you randomized people into each group and then you run your tests and then you have your experimental design research process and it’s generalizable because by randomizing you control for a lot of things all these extraneous factors that you wouldn’t necessarily do with a correlational because correlational is dying a lot of times when you sample for those they’re not a you’re not randomizing you’re just taking people as they come it’s a convenient sample so we’ll do sampling another day so under the three broad classifications we just talked about there are many more design research processs that will fit specific design research process problems or study purposes and these can often be split into two groups of design research process design research process and actually qualitative and quantitative I wouldn’t really call them design research process I would call them more of a method but there are more of an a method of inquiry okay so how are we going to set up our study are we going to ask a lot of questions or are we going to actually collect some data and crunch some numbers and that’s the big difference between two two a qualitative design research processs are used to address the why questions because why why means you you’re going to ask a little bit deeper you know you you’re going to get thoughts from people you’re going to get ideas from people maybe do some interviews and you’re going to uncover some opinions some common methods of qualitative design research processs include focus groups you can have individual interviews just observation you go out in the field and you do some observation and take some field notes ok the data is usually comprised of interview responses or like I was saying field notes so it’s a usually these are long narratives you know not a lot of statistics ok then there’s quantitative design research processs ok and they’re used to quantify the problem and they are you will collect your numerical data or you will collect maybe some data and you transform it into numbers so that you can run some actual statistical tests quantitative data collections methods include all the fun things the statisticians love so we would use surveys maybe short question interviews big huge longitudinal data sets sometimes systematic tests and measurements you know maybe if you’re running a clinical trial you’re checking cholesterol levels things like that that’s quantitative measurements things that you’ve measured and you assign numbers to basically and here are a very short list of some types of design research process design research processs and I put them into two columns so you have some qualitative ones on the left side you have some quantitative design research processs on the right side if you look at the qualitative action design research process that’s kind of close to a quantitative because what you’re going to do is you’re going to a lot of times do some kind of implementation of some kind of new program or some new process of doing things and you’re going to kind of follow it and see if there’s any changes over time case studies that could be one person you’re following maybe a particular case some kind of rare maybe a person that has some kind of rare condition or a case could be it doesn’t have to be a person it could be a program it could be a group of people narrative basically as what it is narrative grounded theory is that’s a really really interesting one what you’re doing is you’re setting up your study you’re taking a lot of commentary information collecting a lot of evidence and then you’re trying to come up with a theory and that’s that’s a very very involved study phenomena lotta feminum I can never say that word knowledge that’s lived experiences of people what you’ll do that this is this is one of my favorite ones and this is the one that you’ll see quite a bit you will maybe take ten people interview them with a set of maybe ten interview questions or maybe even a focus group and ask them some questions and then you’ll take their lived experiences and beliefs and then you’ll build your study around that historical I’ve never see that too much but it’s basically you take historical information and you archival information maybe and you build a study around it and ethnography is where you actually might go and follow some people around you’re kind of involved in their environment and you take your information that’s a very big observational kind of study that you would do quantitative we have our description descriptive studies where it’s just basically straight numbers you have cross sectional cross sectional what that really means is that you’re taking information from people at one point in time you’ll see this a lot when you take a survey you’re going to go out and you’re going to survey some people well maybe you’re going to survey survey a group of people that come into the study over a month or two but that’s it so it’s cross section or you’re just taking that cross section of people at that certain point in time and you’re you’re taking a survey and that’s what and then you’re coming up with your information quasi-experimental is not experimental it’s instead of randomizing people into two groups like you would in a true experimental study what you would be doing is they come in already already assigned more or less the quasi experimental and causal comparative are very much the same and an example would be if you’re looking at gender well you’re already a male or female so you can’t be randomly assigned so it’s experimental except for the point that the groups are already there meta-analysis is very very involved very interesting but what you do is you take all the literature that you can find us out there that serves your problem and your purpose and you sort through it and you try to find this overarching kind of means or standard deviations or some kind of statistics that represent the the literature that you’ve looked at cohort studies and longitudinal studies are very very much they deal with larger databases usually cohorts might be if you’re in a school system and you’re looking at some kind of you’re looking at groups of students over three years but they’re not necessarily the same students okay so they might be students come and go right they they enroll they they move away so you would probably call them cohorts because they’re not necessarily the same students they’re just different different years they’re in the same program they’re in the same school but they might be different students cohort in searly cohorts longitudinal though are the same people so you’ll look at the same people over three years or you look at very large longitudinal studies maybe look over ten years twenty years of course people drop out and such they you know there’s attrition over time but basically the difference between cohort and longitudinal is long a two denotes that you follow the same people I have a note here that a design research process design research process in principle can make use of any type of data collection method so whatever method you want to use of course some design research processs work well with certain methods and you see them more often with certain methods however they’re really in theory isn’t any methods that says you can only use it with a certain design research process so again I said think of design research process design research process as the logic and methods as the logistics right your tools and how you’re going to do it okay so let’s uh we’re going to follow a very simple study as I talk about this a little bit more it’ll help you to maybe understand it a little bit more and get that that that depth that only you can do in an hour-long webinar but we’ll do our vest here so the steps to design research process a very simple study so my study is I want to understand your relationship between sunrise and crowing roosters okay and we kind of already know that there’s a relationship but we’re gonna pretend that oh maybe we we’re gonna pretend we have an idea of it and we’re either we’re gonna go out and test it okay so our first step when where we are we’re design research processing the study is what is my design research process question okay so a design research process question defines our choice at a design research process and all of the next steps so we really need to know what thing and that comes from the problem what’s the problem and what’s my question so here’s depending on our different design research processs here’s some different ways we could ask our question okay so if we’re doing a descriptive design research process we’re going to do maybe a qualitative kind of method so why does a rooster crow at sunrise so we want to get some in-depth information on why why is that rooster crowing at sunrise correlational design research process we would ask is there an association between the Sun rising and roosters crowing you know is there something going on if that there’s an association experimental design research process would be does the Sun Rise cause roosters to crow okay so as you can see there’s a little bit of a difference there and you’ll see how we set them up according to the questions step two which might be more important in step one and now we’re going to start going into our methods territory remember I said you’re going to kind of we’re going to kind of slide into it so now we have our design research process question so we’re going to go from our logic into our logistics so my suggestion would be to start your literature review before you begin outlining your design research process why because you don’t want to do something that other people have done over and over again yeah now typically you don’t I’m all for reproducibility and I would love to see more studies out there that people do that have been done before and it’s just basically we’re trying it again to see if we come out with the same results I think that’s needed very very much in in in all fields but it’s very hard to get published if you don’t have something original or our little tweak or if your dissertation students are always looking for that gap so it’s it’s a shame okay however if you have a good argument for doing a reproducible study that’s great but you might not get published so the good thing to know is have others ask that similar question and have they focused on a similar concept or theory if it’s exactly the same as what you want to do you might want to change it up a little bit what instruments population sample sizes types of statistical analysis have been used in similar studies that literature review is going to save you a lot of effort because you’re going to be citing some things you know this for this study did this and this study did that and that’s why I’m doing this right so you always want to have something to back you back it up also it’s if if the tool is good if you see an instrument that’s great that you find it from your literature review from similar studies any questions on anything so far yeah we have Terry that asks does longitudinal have to do with people can you do longitudinal with companies over years yes you can I’ve actually done studies where I’ve done countries there’s one I have published in fact I think it might be one of the first ones I had published way way long ago ten years ago or so and it was a cell phone use and technology use and a number of protests and it was done over time and over many years over and I think we looked at about 20-25 countries to see if the numbered protests increased over time so yes you can you can definitely do it your units of measure don’t really matter so much it doesn’t have to be people okay so we’ve done a literature review we we have our design research process questions and so now we want to know how exactly we’re going to do it right how are we going to answer this how exactly will I answer my design research process question logistics right the design research process method includes all of the ingredients for our recipe all them or all the materials and work we need to build from that blueprint that we have okay so where it’s going to include data collection how are we going to you know how are we going to collect the data are we going to use surveys are we going to find an archival data set are there some records somewhere we can use are we going to do interviews how are we going to process it how are we going to store it how are we going to clean it up how are we going to code it data.
I always say data comes in it’s ugly it’s mean it’s nasty it’s dirty it always needs cleaned up so it’s always nice to put into your into your methods into your plan how are you going to code it are you going to call code male zero females one if you do a regression that would be the way to do it if you do it a Nova model you might code them one and two so really that these are things that you really want to think through and think out you how are you going to summarize the data you know how are you a lot of times in a statistical analysis plan for clinical design research process you you have tables and figures you have a list of tables and figures that are kind of dummy tables or mock tables you don’t really have the numbers in them yet you well you have made-up numbers but it kind of it’s great because you can take a look at those tables and it helps you to think ahead of how I’m going to structure this how am I going to show my information and things always change you know your best laid plans so keep that in mind too however it’s always good to start with the plan and then you just talk about what your deviations were once you collect the data and clean it up and you know find some new new problems to deal with but then you also want to talk how are you going to analyze data what kind of tests are you going to use and then of course you want to how do you plan on interpreting the findings you know or what are you looking for and and how do you want to maybe present that so over here on the right hand side the statistical analysis plan is often what we use in a clinical design research process to detail this design research process and Method in a high in a dissertation your methods are usually your third chapter and your results are usually your fourth chapter so we’re in methods territory now we’re planning it we haven’t analyzed it yet so those that would be some things we need to think about now we’re talking about methods what kind of information can you get okay definitely important to know it’s great to have a great idea and that you want to study something you have your problem you have your design research process question but you have no way of getting that information then then you won’t be able to do the study so again the literature reviews great what tools and instruments are available if you’re going to collect data with a survey or are you going to use a database if you do use your own survey or if you use a survey let me put it this way if you use a survey find one it’s out there get permission from the author to use it usually they’re very very happy to let you use it as long as you acknowledge them and you know give them proper proper credit in your work some some surveys are proprietary and they want you to pay for them which you know it can be done but often times you can find one that’s free to use or you can get authors permission so you know feel free to reach out to the people that have developed these tools and tell them about your study and what you plan on doing and they’re usually more than happy to let you use it or even adapt it you know and just let them know you plan on adopting it and how you plan on doing that and then if you do adapt it you want to there’s there’s some checks and such you would want to do but that’s again another webinar but maybe one day we’ll do one on validity and reliability but again if you develop your own tool you’re going to have to do all these checks yourself dissertations are done on developing tools but if you want to do an actual design research process try to avoid developing your own tool unless you absolutely must do so it’s going to save you a lot of time a lot of headaches okay so what kind of information can I get we’re still thinking about this how are we going to review and analyze that data are we going to have hypotheses to test if you have a qualitative study typically you only have design research process questions you don’t test hypotheses hypothesis test of course you have to have a statistical test right so you’re going to use more of a quantitative kind of design research process so that’d be something to to to Figg think about am I going to be collecting data and testing hypotheses with a statistical test and what test should I use and boy this is a this is a big question the options are many here’s an example I stole from Penn State stat 500 website these are all different quantitative tests there’s a place where do you start let’s see I think we would start right here.
Do you want to estimate a value test a hypothesis examine a relationship so you go from here do you want to try to find maybe a mean or something go more of a descriptive route here we go we’ll follow these threads here if you want to examine a relationship now we’re looking at okay this is more of maybe a that correlational design research process right so that we have quantitative or categorical variables quantitative me basically are like a continuous type of variable.
Hi wait time.
BMI things like that categorical are are just like it sounds or categorical they’re also called nominal you know ethnicity gender things that can’t be ordered some people will say that ordinal are are categorical also so again these the one thing about statistics and I always joke about one of my favorite one of my favorite things to say statisticians have three three to five names for every concept there is and and the reason that is is because we do work in so many different backyards and so many different fields of study and each field of study has their own way of saying and looking at things their own nomenclature so it can get really really crazy but basically you you need to think about this if you’re going to be using statistics and testing the hypotheses you know what what are some of the what way should I go with this you know a t-test would compare two groups and a Nova would compare more than three groups and it’s comparing on mean scores so if you want to compare medians there’s different tests so again boy this is this is many many webinars here however you need to figure that out how am I going to test this what analysis am I going to use so I can spend a little bit of time on this if anybody has any questions no questions okay then he come to you just let me know no problem okay okay so step 5 we’re going to develop our hypotheses we decided we needed them okay we’re using a quantitative study I want some hypotheses so these are statements that you make the test your design research process question okay so we word them to fit the tools that we’re going to be using so if we’re using for instance correlational design research process we would at you know in our hypotheses we would say something like is there or basically it’s it’s not is there’s not a question it’s a statement so you would say for the null hypothesis there’s no difference between gender and well-being scores okay or your alternative hypothesis would be there is an association between gender and well-being scores if you had a comparative study than it would be do males and females different on the mean of their well-being scores okay so now we were asking if there’s differences and you’re doing a comparison so definitely depending on your test your hypotheses are going to be different the null hypothesis means no effect null means none nada so this is the hypothesis that states there’s no difference that’s the hypothesis we’re testing we’re testing for no difference okay the alternative hypothesis is what we want right it’s what we want to see we want to see a difference between genders maybe on well-being or we want to see an association between roosters crowing and the Sun rising so that’s going to be all our alternative and as I’m saying down here at the bottom it’s the confusing thing is that we will draw our confusions our confusions yes we’re going to draw our conclusions too and our confusions based on the null hypothesis I’ll tell you when I was an undergrad I don’t think it was until my junior year that I’d really really wrap my head around that we test the null hypothesis is it just drove me crazy that it just didn’t make sense but we’ll talk about a little bit more later I’m going to go through two three different design research processs and kind of show you how we set them up these questions so let’s start with the descriptive design research process okay like I was saying this is mostly your qualitative kind of thing you can use it in a quantitative study but typically descriptive I always associated with the qualitative.
We’re going to get that why we’re trying to get that depth of meaning we’re trying to dig down deep so it consists of only the design research process question no hypotheses and it’s because it’s descriptive and we’re not testing anything right so descriptive studies can be they can be both quantitative and qualitative or quantitative an example.
I gave that naturalistic observational study and a design research processer would ask what proportions of roosters crow at sunrise and then she would visit a farm each morning at sunrise and she’d count them maybe Talia every time the roosters crow she’d make a little check mark and perhaps she notes the time they crow how long they crow and and she might she might also she might want to control for weather conditions maybe if it’s sunny out they crow more right if it’s cloudy out it might be lighter out but maybe they don’t crow as much maybe that the brightness of the Sun might have some kind of effect so that might be something you want to control for in your study for qualitative there’s many types of qualitative studies and I’m saying here too many to cover individually but in general it’s an observational study and the goal of obtaining a deeper understanding so the design research process would ask why does a rooster crow at sunrise that’s so philosophical isn’t it why do they why do they crow and she can’t interview the roosters but she can ask a group of chicken farmers maybe she has ten open-ended questions and she’s going to ask them about their thoughts and beliefs on the phenomena so her studies actually would be about the the farmers perceptions of why roosters crow not actually strictly about the roosters so she’s adding in that element of the farmers okay that’s pretty cut-and-dry correlational correlational design research process i’m usually have the words relationship or association predict is often used if there’s a regression you’ll have predictors and you’ll have criterion and criterion is your dependent variable or you’re out again we have to have three three three names at least for everything right so in a regression model the dependent variable is also two criterion and it’s also the outcome okay probably another name or two also but our design research process question for a correlational design research process on the roosters and sunrise would be is there an association between the Sun rising and roosters crowing they mine all hypothesis is going to be there’s not a statistically significant correlation between the sun rising and roosters crowing the alternative would be there is a statistically significant correlation between them so my design research processer now she’s measuring if the Sun rises each day okay and she’s going to code it see because she’s going to operationalize things and this is part of the methods you want to make sure you detail it as much as you can she’s going to code it one if the Sun rises and she’s going to code today at 0 4 if it’s cloudy rainy anything where it doesn’t rise and shine okay the design research processers also going to measure if the roosters crow within a five minute time interval after the scheduled sunrise and then she’s going to also code those each roosters a one for yes a zero for no so she’s true sir she’s going to see if they crow and then she’s going to get her sunrise schedule from the Farmers Almanac right so I mean it’s definitely going to be much more detailed than this but as you can see it kind of gives you an idea of a lot of times when you’re design research processing a study you ask a question and it just makes you ask five more okay so it’s like okay I want to test the association between these two things how am I going to do it what tool am I going to use how am I going to code it you know what kind of metrics am I going to use you know what kind of if you’re doing some kind of clinical design research process and you’re using some instruments how are they calibrated you know I have they been validated.
There’s so many things you want to check into you know you know what’s the what’s the standard that people use that’s also a good one you know if there’s a standard reference like a farm Old Farmer’s Almanac do you have to make adjustments for your time zone right so lots of things to think about but that would be how you would maybe set it up for a correlational design research process so the design research processer she’s going to perform before more correlational analysis in this case we’re going to make it a fee correlation because the both variable responses are dichotomous yes or no okay and no was coded as zero one was coded as yes so design research processer finds a strong and direct correlation between sunrise and roosters crowing okay so strong and direct means that they’re they’re very much associated and when the Sun rises the roosters crow when the Sun doesn’t rise the roosters don’t crow so they’re kind of moving in a like manner okay but wait remember what your intro to statistics professor always said to never forget correlation does not imply causation right the Sun and a rooster correlational study.
I love this I think this is a great concept for trying to understand the correlation and causation so you might have significant correlation and boy there’s an association let’s say that it was a correlation of 0.8 which is very very strong to p-value is less than 0.05 so it’s significant we’re happy happy great correlational Association everyone knows the sunrise is related to rooster crowing everybody knows that however the roosters crowing did not cause the Sun to rise okay therefore you can’t say that there that they did that however everyone knows the sunrise doesn’t cause a rooster to crow but that it does right it does cause a rooster to crow and that can be tested with an experimental design research process which I’ll show you next but any questions on the correlation causation thing I know sometimes people a lot of people get confused on that we have Jesse asking so when you don’t really know this is basically how you write the hypothesis or if you have a guess do you phrase it differently then it goes on don’t really know what the answer might be yeah you don’t if you don’t know what the answer might be now you’re going into it an exploratory study and and something like that.
I like to refer to them sometimes it’s fishing expeditions you know because you’re going and you say maybe I have some data here I just want to look at it and see what’s going to happen you know if I’ll look at it this way now I look at it that way you know so you really do want to have a design research process in a plan to really keep it a good scientific type of study in accordance with the scientific method that means that you need to have some kind of a hypothesis you need to typically whenever you’re doing a study you already have an idea in your mind of what you think would happen and that’s what you’re testing otherwise you’re in exploratory yeah typically with a scientific method you’re going to have your alternative hypothesis is basically going to be what your thinking is going to happen and your null is saying it’s not going to happen right so it’s basically it’s this this I wouldn’t say trade off but it’s almost like a you know like a a tug-of-war right you have the null hypothesis saying you’re testing for it not to be a no effect you’re testing for not to be seen but you’re really hoping that it is and so you do go into most scientific design research process you’re going to go into it with some kind of a priori idea of what you think is going to happen so yeah so correlation does not imply causation so if you want to do something that’s going to show some kind of causation you need to do experimental design research process okay so how does one show what we already know in this case that the Sun causes the Roosters to crow right we know this so you wouldn’t really be testing for this but like I said I’m keeping it simple so I thought this would be a good example the experimental design research processs the gold standard okay it’s the only type of design research process in which one can claim cause-and-effect an experimental design research process it has internal validity it’s at its core which means that if a manipulation is performed then the outcome happens and if a manipulation is not performed then the outcome does not happen.
I’m going to talk a little bit right here about deterministic and probabilistic type of approaches to to to logic to a problem deterministic is basically that if a then B and always if a then B okay if a then B then C always that way right so if they’re in and and then a goes to B goes to C that means B still goes to see it’s very deterministic it’s set in stone design research process isn’t like that the the world isn’t usually like that very very rarely are you going to find something that’s that logically in step so we have probabilistic and that’s where you’re going to take into consideration yes maybe with this group of people it’s true but knightmon might not be true for other groups of people might not even be true for a different sample okay so that’s why we always have to test and retest and you find some conclusions in one study the next study they might not hold up okay that’s whole part of the scientific method and there’s a paper at the end of this I’ll refer you to that you can read a little bit more about deterministic versus probabilistic but if it was that as easy as one two three step step step in everything in the world I probably wouldn’t need to do as much as I do okay so but for an experimental design research process if you randomized things into random I say things if you randomized people for instance into two groups treatment versus a placebo okay one one’s going to get the treatment the other one’s going to get some kind of sham or placebo what if that does with the randomization as they come into the study into the two groups is that it helps you to control for anything that might be different any of those probabilistic outcomes where because it’s not deterministic because it’s not set in stone so it helps you control for every other thing that you can think of or things that you haven’t thought of and and it kind of washes everything out you’re randomizing so it kind of takes all those all those differences out of the picture so you can just look at the two different treatments or a treatment placebo and then check for those effects it’s going to make it so the effects or for what you’re testing for and that’s why an experimental design research process is is the gold standard okay so as I’m saying here the outcome could be an improvement and change in health it could be test scores it could be a an appetite any other measurable factor observed when a treatment or interventions performed on one group but not on the other group okay and there can be more than two groups but we’re only going to use two in our example okay one caveat is that the two groups must be as similar as possible prior to the treatment or intervention and that’s why we do the random sampling that helps to even a playing field for everybody into two groups okay and again randomization is such as maybe something for another webinar we can talk a little bit more about it so here’s our design research process question does the Sun Rise cause Roosters to crow and our null hypothesis is that there’s not a statistically significant difference in crowing frequency between Roosters who see the Sun Rise and roosters who do not see the Sun Rise so we’re going to say there’s no difference the Sun doesn’t make any difference or either you’re gonna take row the same amount or they you know we’re assuming they’re gonna crow because grew stirs crow right we’re going to basically think okay well they you know the the ones that see the Sun don’t crow any more than the ones that don’t okay the alternative would be that there is a statistically significant difference in the frequency of crowing between Roosters who see the Sun Rise and roosters who do not see the Sun Rise here if I have some really really picky statistics people in the audience today this is a two-sided hypotheses right you might want to make it a one-sided where you would say all those Roosters who see the Sun crow more significantly more than the other ones and then you would have a different type of one-sided hypotheses but on this we’re just going to say is there a difference we just want to know if there’s any difference okay so we’re going to test for that so here I’m going to have some inclusion and exclusion criteria and actually we would want them for any study you want to think about who are you going to include and are there some things or some types of people or some types of sampling units that you don’t want to include and why so for your inclusion we’re going to have male chickens okay because they they’re the ones a crow and they’re going to be over one-year-old okay and they have good eyesight so they can see the sunrise and there and they have a dumb straighted ability to crow we know they’re able there is there’s not something wrong with them where they can’t crow I don’t know how we’re going to test their eyesight that would be something to think about but yeah so we want to make sure that they can see the Sun right so these are things they like I said it’s kind of silly but it kind of gets the point across and that’s why I chose this one but that would be something that maybe I’ll think about that this afternoon how would I test the roosters roosters eyesight so the exclusion would be female chickens right because they they might crow a little bit I they’re not going to crow like a rooster just son at least we don’t think they will chickens less than a year of age right little Pete so we don’t want little peeps and we don’t want roosters who aren’t don’t don’t see well again I don’t know how we’re going to test that but we’ll figure that out and they’re like I said they’re important to all design research process design research processs okay so always make sure you have that in there so the design research processer recruits some roosters who meet the eligibility criteria from various farms located in the area has them sign a release now she doesn’t but maybe she has to check with the farmers before she goes and recruits the the rooster so she’ll have them sign a release as saying she’s allowed to go and collect some roosters from them and as each rooster comes into the study she’s going to randomize them she’s going to put them in the either in an intervention or control group so the intervention basically is going to be the sunrise group controls are going to be the ones who don’t see the Sun so she’s going to randomize them so she’s going to get her roosters and she’s going to just says they come in she has a randomization schedule and she goes okay this roosters going to go to group a this was the next roosters going to group a the next groups going to be and it’s not a b a.b a.b okay it depends on what your randomization schema says and so she you’re going to go ahead and allocate them in so so the intervention group are the ones that are going to see the sun so during the evening they’re going to be placed into a pen with an opening on the top and it’s going to allow the Sun to shine through in the morning okay and then for the control group during the evening she’s going to put the roosters to sleep into a pen without an opening on a top no Sun is going to shine through in the morning okay and except for the pen assignment in the evening whether they’re going to go into the open top or the closed top the roosters are free to go about their day and there I guess in a communal area I put them they like they get to hang out together they get to eat the same food they’re going to exercise the same amount you know so you’re not going to have some in a bigger pen that out running around free-range and the other ones in little cages you don’t want to do that you want them basically to have everything the same except for that pen assignment in the evening okay and you want them to have the same social interactions things like that.
I can’t believe I sat and thought this has kind of fun though so the design research processer could have exerted much more control in the study and just placed one group a Rooster’s in the sunny pen and the other group in the dark pen 24 hours a day but she’s a kind design research processer an ethical the animals so yes that’s a limitation of the study is that she didn’t confine them 24 hours a day for like a week into their pens and a limitation you know it is a limitation because maybe maybe a rooster gets out and runs away maybe you know maybe there’s there are some there’s some things that can happen when they’re out and about in the daytime that some of them and not the others you know so it is kind of messing with your strict randomization a little bit but all design research process is give and take and you really have to think about ethical ethical considerations especially in clinical design research process so the patient matters the most okay so you want their comfort you want them to you don’t want to hurt them in any way physically emotionally you don’t want to cause them any kind of stress so sometimes there are some limitations you have to put into your you have to allow your study have limitations so that you’re not doing draconian things to people or roosters okay so each morning the design research processer notes the number of roosters who crowed in each of the two groups during the ten minutes after sunrise again she’s computed that from her Farmers Almanac so she has checks in place to make sure she accurately counts all of their crows this is going to be hard to do because she has two pens of crows right so she might need a helper okay she also might want to audiotape okay as she’s doing it just as a backup so it’s good that you know again put these kind of things in your plan think it through be holistic about it the thing about when you’re doing these kind of things to time and money okay there’s there’s the time and and there’s a temporal and monetary cost of running in a study it’s a big deal okay so simple tape recorder a tape recorder okay.
I guess audio recorder now I guess we don’t have tape anymore I’m showing my aides that the simple tape recorder could be less expensive than a helper right but using a recorder can add time to the process because now the design research processer has to listen to the recording match up her tallies to her croak outs and she can have both she could have a helper and a recorder but but again you kind of need to think about time and money because you don’t want to go too far with all the checks and balances like I said all studies have limitations and you’re just going to have to accept that you know that you don’t want to have glaring limitations that are going to invalidate things but you know it might be that you know and in this case a helper and a recorder that’s fine but you don’t really do you need five people out there probably not okay so after one month oh boy she did it for a month okay so she compares to a number of crows that were emitted at sunrise for each of the two groups or roosters okay and she compares these findings okay she determines that the Roosters that saw the sunrise crows significantly more than the Roosters who did not see the sunrise and now she has an experimental study with good conclusions and it’s generalizable okay and let’s revisit that so since her design research process was randomized and controlled she can potentially generalize her findings to similar types of roosters that did not come from the farms where she collected her current sample okay it must be randomized in order to even attempt to claim generalizability and it might not be so easy okay what if the design research processer only used one breeder rooster okay could defining to be different if she sampled from farms in another part of the world and this is where repeatability is going to come in handy it’s important to make sure you cover all the bases you can okay it seems overwhelming think of it think of the big picture but do it in little steps right let’s step by step okay so more about methods methods are the roadmap of repeatability operationalize your variables how are you going to code them are you going to call code males as one and females is two are you going to code them zero one how are you going to code ethnicity are you going to use dummy codes where you each ethnicity is a variable or are you going to make them you know white one african-american two Asian three how are you going to do that you know so it and that depends on the test you’re going to use okay so these are these but it’s always good to get those operationalize get an idea when you’re doing that when you’re thinking it through like this it’s going to help you to see some problems and like I said for each question you asked you might have five more but these are questions that you want to think through before you pull your data and have an Ono moment where you go oh no I should have right you don’t want to do that you’re going to do it probably anyway because no studies perfect or something will happen that you didn’t plan for or you could have planned for it and something totally different that you would never even see happening could happen so but you want to minimize those Ono moments right so yeah determine those analyses have a back-up plan what if you want to use say an ANOVA and you have your normality is not really a big deal they’ll tell you that in school but say your variances aren’t equal okay that’s a big deal are you going to transform your data what if transformation doesn’t work what if it makes it worse hmm maybe there’s a nonparametric technique you could use where you don’t need that normal distribution you don’t really need two equal variances may be a kruskal-wallis test so you would you want to have that backup right kind of you have a plan B okay think about possible control variables okay so the big five for many studies are age gender race ethnicity SES s which is socioeconomic status and marital status of course roosters might have other control control criteria right maybe age maybe breed so maybe you want to put into your model the age of the Roosters maybe control for those maybe if you did have different breeds you want to make a variable breed type you know or maybe a few dumped probably it depends on your study you know you but you might want to incorporate the breed of rooster in there because maybe some just maybe some breed or roosters they just Chrome or maybe some breeds of roosters don’t grow much at all I don’t know much about roosters but that’d be something to think about something to search the literature about something to ask the farmers about right so there are numerous methods for collecting data and crunching numbers okay so for the most part methods of sampling an analysis they’re interchangeable between qualitative studies and I think I covered that before so a certain type of study a certain like a causal comparative you you could use many different types of tools you can even use a survey you don’t have to just use that for correlational okay here are some references the first one is organizing your social sciences design research process paper which is it’s a very good one and it has some different different quantitative I think it’s mostly quantitative design research processs it kind of gives you a little bit of information on it there’s a there’s a paper on experimentation which is very informative the third one that’s the one I was talking about where deterministic versus probabilistic and it’s the intricacies of design research process design research process very good paper to read I would suggest reading that and then some good books by tabachnikov Fidel on experimental design research process and multivariate statistics which probably it has all kinds of different methods and design research processs in there so some very very good references there some I’m basically done so thank you for attending here’s all of my contact information I will be sending a handouts and I will be sending this recording out as soon as mr. rod Eisenbeis puts together the nice pretty recording takes out all of my pauses and you can leave a couple of giggles in rod and don’t clean it up too much but I’ll get that sent off and I think the next webinar is going to be in February I have them every quarter I think the next one I’m going to do is on power analysis but and that’s for sample size because I think that would be the next step for all of us that are following along here we’ve done the literature review now we’ve done some design research process and methods information let’s talk about how big of a sample you need next.
I think in February and of course if you have any ideas love to hear it I would love to do some webinars for what you guys want because that’s why I’m doing it right to help out here to give you guys some information and so that is it for the presentation so I’m open to questions and boy we have we have some time so if anybody has any questions let’s let’s let’s ask one question is how important is it to have a random sample if you want to talk about generalizability it’s very important and and it doesn’t this you don’t necessarily have to pull the sample randomly there’s random samples and then there’s randomization random sample is basically you take a list of your population say you have a list of people let’s make it easy in a geographic area say I’m in Marietta California so I have a list of everybody that lives in Marietta and well it’s not we don’t want everybody let’s just have some inclusion criteria right.
I just want all the women that are between 30 and 40 in Marietta and I’m going to ask them some questions on a survey okay so or no I’m going now that’s correlational design research process let’s say I’m going to ask them I’m going to have them try some kind of program maybe an exercise program okay there are 30 to 40 years old they live in Marietta I’m going to try the myriad a diet plan okay so I’m going to take them off this list and I’m just going to randomize them off a list right I’m just well I’m just going to randomly sample them from a list and I’m just going to go okay you know the ones that are on my schema here I’m just going to pull them off the list I’m not even going to tell them not it’s not even going to ask if they want to be in the study they’re just randomized show take the diet or not right you know so you’re going to be one or the other automatically randomized that usually doesn’t happen and the work I do a lot of times I do a lot of cancer design research process and the the patient already knows they have cancer they’re coming into the study and are going to be randomized okay so you can’t go you can’t you know give people you know a disease you know so you have to take them as they come and then you randomized them into the treatment or the the type of treatments are going to happen a lot of times especially with the cancer studies they have a preference for treatment you have to take that into account also so that kind of messes up your randomization so you will have some people that they don’t care they’re happy to be randomized and you have some people that really care or maybe their doctors care they want them to have this certain treatment so you have to then those patients you can’t use in your generalizability you can only use the ones that agreed to be randomized so it gets very complicated but those are some some examples Martin asks if experimental experimental design research process is gold standard but not possible in certain situations are there steps design research processers can take to make correlational design research processs more robust if not as robust as experimental um if you’re talking in terms of generalizability which is what everybody wants to do you know like oh I did a correlational study and not only is it good for the students in the school that I looked at oh all of ninth grade students in a state you know will this this will apply to the only you can’t really do that with a correlational type of study or any kind of convenient sample kind of study right so what you the best thing you can do is that repeatability have that study repeated over and over with different groups of students in different schools you know maybe you have the ninth graders in this school and this state and maybe now we want to go to California and test the ninth graders in this certain school you know maybe a similar school it’s just the only way you can really really get that to be more generalizable is just with three.
PETA testing Jesse asks do you have tips for finding existing survey instruments when you know you are doing a survey and martin says thank you oh you’re welcome okay so tips for findings particular surveys existing survey instruments when you know you are doing your survey okay there are some Google Google’s your friend first of all the literature review when you’re doing your literature review because first of all like I said you don’t want to do a study that’s already been done before so and you also want to see what has been done before so your first step is that literature review and as you do that and you look through your journal articles you’re going to see what instruments they might have used okay the other thing you can do is Google and you know emotional intelligence for instance I had one I had to try to find emotional intelligence actually soft skills which is very closely related to emotional intelligence so I looked up a Google soft skills emotional intelligence things like that and I kind of narrowed it down to find a couple tools that way but really the literature Google and you know just maybe look at PubMed some of the databases and and maybe try to again look at the literature for some tools if you are in a field where your colleagues are very knowledgeable of the type of design research process you’re doing they might have some ideas so it’s basically you got a design research process right you got to be a detective and try to find it that way as far as there’s databases all over the place and you can just type in I mean there’s depending on what what your study is about a lot of times that the data has been collected in large database you know there’s an Haines or see all kinds of CDC databases I found one on Ebola I’m working on an Ebola study that the world World Health Organization had so some of these big organizations Census Bureau they keep data and a lot of times you can get some permissions to use the data but yeah for instruments try not to design research process one yourself try to find one you can adapt and Google and the literature is probably your best bet Jesse asks but you don’t have to do the full validation step if you find something it depends if it’s a tried and true Rosenberg self-esteem scale been used for years it’s been used in thousands of studies the only thing I would probably really I wouldn’t check validity with that because it’s been validated so much but of course when you’re writing your methods you want to talk about all the studies where it’s been not every single one but many of the studies where it’s been validated and studies that are similar to the ones that you’re using right if it’s as if it’s a survey you’re using it’s never been used in a study like yours then definitely you want to try to do some more checks on it typically though whenever I get a survey that’s been tried and true when it comes time to do the analysis I’ll do if it’s a like a Likert kind of scale I’ll do chromebox alpha and that helps with the internal consistency reliability and what that means the chromebox alpha an internal consistency reliability means that you’re testing to see if that survey holds up with the sample of people you click with their answers right so you say you pull a sample of a hundred people you’re using Rosenberg self-esteem scale and so you want to do a chromebox alpha to make sure that each person is kind of if they’re if they’re have high esteem that they’re answering kind of on a high across the board if it’s a person with low self-esteem you would expect a lot of ones and twos lower scores and so you want to make sure that each person is consistent and that’s what chromebox alpha does and that kind of that’s a test of reliability but if it’s a tried-and-true instrument the validity checks can you know the divergent and convergent validity all of that is usually done and that’s why I say try to get a tried-and-true one don’t try to develop it yourself because otherwise yeah then you got to do your validity reliability you know the convergent divergent all that it’s it’s it gets crazy test retest pilot you know so does that answer Jessie for you Terry yeah Jesse didn’t respond by them Terry asks so did you say you use design research process questions when you do qualitative design research process but hypothesis when you do quantitative design research process it seems like they are mixed in the literature review typically that’s the way it is Terry yes you would use only design research process questions for a qualitative okay because the hypothesis statistical hypothesis their test you’re testing hypotheses right so that means you’re going to be using a test you’re going to use a t-test they you know of a regression any number of statistical tests so therefore you need a hypotheses statistical tests are usually done with data so you have a quantitative study that’s using data to test hypotheses with an actual test qualitative on the other hand a lot of times it’s just interviews maybe you talk to ten people about their thoughts on a new program on TV or something you know it won’t be like that it would be something more more than that but say you have a group of ten people there in a nighty apart an IT department and they have a new computer system what do they like about it what don’t they like about it what do you know how do they feel about it you know why that’s a qualitative type of study so you’re not testing anything you’re not using a statistical test you’re just giving some descriptive information a lot of narrative and coming into some thumb attic some themes like oh they really like the you know the user interface they they hated the colors you know things like that so no you wouldn’t need a hypothesis for qualitative there are mixed methods studies where you have both where you will have a qualitative element and then maybe you will do some hypothesis testing of some kind of a you know of another element of it then yes they would you would have both that’s called a mixed methods design research process when you when you put both together but typically yes qualitative design research process question quantitative design research process question and hypotheses because you have a test Terry says perfect that makes it much much clearer when I see a I’m sorry when I see a hip my hypothesis versus not another question is how big of a sample do I need what is a sufficiently large sample that’s a very good question and it’s so dependent on so many things I’m going to I am going to do in February I will do the power analysis and it that’s what it’s about sample size and so we’ll cover it then but it depends on the test you’re using if you’re using the you know and it depends on how big of a difference you’re looking for between groups or what your effect is there’s something called an effect size it’s like it’s like how big of a difference are you looking for or how big of an association right if you want to vary if you’re testing to have a very strong association between two variables then you know you’re going to need a different sample size and if you’re testing to maybe say that oh if I have a correlation of like point two you’re going to need a lot bigger sample you know so typically if you’re looking for big big differences you need smaller samples if you’re looking for teeny-tiny little differences between groups like say a mean difference of two points on a 100-point scale then you’re going to need a pretty large sample so it’s all its varied there’s three actually there’s four things that are involved in getting the sample size it’s a there’s a pneumonic it’s called bean let me see if I can remember you have to have your B is for your for your beta which is your power you’re going to need so typically that’s like a point eight eighty percent e is for your effect size how big of a difference do you want to see a is your alpha level which is usually that point oh five that p-value you’re looking for which is 0.05 and n is your sample size so if you know three you can figure out the last one so in the power analysis webinar and that actually I do also have a power analysis on youtube channel or on the website if you go to the website the webinar page there is there’s an older one of power analysis look that / and that’ll give you some idea and then we’ll we’ll do it again in February and answer some more questions and get maybe a little dig a little deeper into it another question is can I have more than one design research process in my study what if I have more than one design research process design research process question you can’t have yes you can have more than one design research process you can have both the qualitative and the qualitative and quantitative methods so you can typically you can’t have more than one design research process but I typically would say try not to try because here’s the thing you’ll as you go through your design research process as you set things up you’re going to probably have to make it smaller right you’re probably when you first start out you have big ideas I’m going to test everybody in the United States to see how they feel about you know see see how relationships how marriages that have been around for 20 years how what are some of the things that that you know they they keep them together oh my gosh that is so huge right so you want to bring it down to maybe you know people that have been married over 20 years maybe give them a survey of some kind of how well do they know each other something like that and and pare it down a little bit so but yes you you could possibly have different a couple different design research processs you definitely can have more than one design research process question okay but again if you’re doing a dissertation design research process I would probably try to maybe have an overarching design research process question and then have a couple hypotheses to address that design research process question a couple statistical hypotheses in a in a quantitative and a qualitative I’d have an overarching design research process question which is like related to your problem statement and then maybe I might have a couple more design research process questions sub design research process questions or tinier ones are more specific but try to don’t try to make it the end-all be-all design research process just doesn’t hold up well whenever you’re looking at a lot of different things at the same time you want to try to pinpoint it all right and that was the last of the questions okay well we did pretty good had some time to spare well thank you everybody for coming today this is really fun I hope it was informative and I see the people that came stuck stuck it out so high.
I’m glad I’m glad I always loved doing these things and I’ll see you guys in in February and have a great holiday oh my gosh.
II can’t believe it’s November already have a great holiday season and – next year we’ll we’ll keep at it and we’ll move this forward and have a great afternoon everybody.
Thank You rod thank you okay bye bye.
Where to find great research papers?
Various great research journals such as Global Research Letters are a great option and way to help you look up impactful research papers with a great format. Here, you will find a number of various research papers that are provided and made available to you in the journal, which will help you write your own paper.
You can very easily find papers on a variety of topics at Global Research Letters, which will help you with your own research work and understanding of writing and publishing research papers properly. With access to so many amazing research papers, you can practice and learn the process of writing research papers and their importance.