Alright welcome everybody thank you for coming and thank you for showing up actually on time which is a miracle given that this is a new time and people may not have 12 necessarily look very close yet time it's a pleasure to have today with us. Dr. Lindsay McGowan from North Carolina State of course from main campus. She's a homo psychology department. She'll be talking about evaluating the impact of cybersecurity. Science mix mix method approach dr. MacGowan is that a PhD in psychology public interest from ncsu in 2012. She's a scientist in the innovation. Such studies lab in the psychology department. Her research interests are in program evaluation innovation studies and implementation studies. Collaboration consortium development programs such divinity sustainability and public health. She's currently working as a project manager. She's acting as a brother project manager for the innocent industry university co-op Research Center evaluation project and she's also working with the NSA science security lab let here in NC State for those of you this is also connected with the LS lab and she's also teaching classes in action research and innovation technology she serves as a center evaluator for three in. SF u crc Center for Child Injury Prevention Center in Center for sustainability integrated building and site and the center for not oh no mobile I Baltic temperature materials or structure. She's a member of the aea and the RTP regulators Association and the society community research in action and she's well regarded in work. And it's a pleasure for me to introduce her. Thank you thank you so much. So um appreciate the introduction. I was very kind. I wanted to come here and talk to y'all today about evaluating the impact of cybersecurity science so this is based on a project that I am working on with some folks here in a computer science department dr. Lori Williams and dr. minh and are seeing have funding from the NSA to do research on how to make cyber systems more secure and how to be more scientific in the way that we approach these problems and so this research is based on my work with them so um I'll just give you sort of an overview about why it's important to assess the impact of science technology and innovation programs what program evaluation really is and then.

I'll give you this sort of case study of the science of security lab let evaluation that we're doing here. I'll talk about what our evaluation strategy is. I'll show you sort of a logic model that we use to structure that that evaluation and then give you some illustrative examples of data that we've collected in the past year as well as other metrics and methodology that we're working on going forward so I probably don't need to convince you. All science engineering technology innovation is important but it's one of the major drivers of the US economy. We spend billions of dollars on that each year. It's the fastest growing area at the workforce with computers and mathematical sciences and engineering be the largest growing sector and academia is really involved in that were the second largest conductor of science and engineering research conducting about fifteen percent of the national total our indie efforts. And there's significant funding and being provided to do this type of work. So what do we get for all this in effort and all this investment. We're using tax dollars to support programs like this and so it's important to understand what their impact is and so program. Evaluation is just a systematic process for answering that question. What are we getting for all the effort in investment and it may seem pretty clear that the the main two goals of program evaluation are documenting the impact and demonstrating that your effective. But it's also useful in as you go along finding ways to improve so that you are achieving the highest level of effectiveness that you can as well as identifying what works and what doesn't work so that others who are trying to do the same thing can learn from your efforts so this is just an example.

There's lots of program evaluation going on all across the country but this is one it's going on here. NC State so just to give you a little bit of background the science of security lab late' is funded by the NSA. This is their second three-year award and they get about two and a half million dollars of funny each year from the NSA to conduct research there's four goblets there's one here at NC State and then there's three more at the University of Illinois urbana-champaign Carnegie Mellon and University of Maryland and each of those lead sites has other collaborating institutions that they work with and the goal of these lab. 'let's is to conduct research around a set of hard problems so they're focused on five hard problems so human behavior related to cyber systems. How do you users software developers and attackers affect the security of cyber systems. Scalability and composability. So how do you go from one system to protecting multiple connected and networked systems and security policy. So how do you develop rules and procedures for making systems more secure resilient architectures so again when you have systems connected how do those connections how do you protect those connections so that the system continues to operate even under the face of attack and that was for forgetting one so anyway they're working around these five heart problems and they're trying to not only develop solutions to these hard problems but also to identify develop and adopt rigorous methodology around solving these problems so cyber security and computer science in general and there's a relatively new field compared to things like biology or physics and so we're trying to learn as we grow develop new ways and better ways to understand what are the fundamental principles of a secure cyber system and then they're trying to develop the community folks who are focused on these problems in doing research in this way so they're trying to develop a community of practice so those are the three goals that we're attempting to evaluate as based on what our program mission is so you're trying to solve big problems.

You're trying to develop a new scientific approach. You're trying to recruit people into the field that seems like a lot. So how do you evaluate when your multiple goals diffused over a long period of time. You're talking about developing a new scientific approach. That's you know not a three-year goal 10 or 20 year goal and and these are really long range objectives. So how do you do that during a three-year time period of grant funding so this is an evaluation logic model. And this is how you do it. And this is a tool. That's used in lots of different types of evaluation from engineering evaluation community evaluation. Anything it really just allows you to lay out. Here's the problem that we're seeing. Here's the steps so we hope to take to solve that problem and here are some of the sort of things that we hope to see will happen in order to get us to the ultimate impact so just to sort of outline if we have if we have a given problem and we use these resources to conduct these activities this result and these outputs we see these specific outputs we expect them to have a certain outcome. And we expect this outcomes ultimately have these impacts and at each point along the logic model you can ask evaluation questions. So do we understand the situation correctly. Did we have the resources that we need do. We conduct the activities that we thought we should conduct. Did we produce those activities produce the outputs that we thought that they should do those. Outputs have the outcome that they should. And what was the ultimate impact of all this effort. And as you're asking these evaluation questions you're hoping to get to this sort of summative evaluation so based on all of this. This is what we see. That was the ultimate impact of this this program but you can also do as I mentioned before formative evaluation.

And that's looking at. How do we use the feedback that we're getting all along the way to improve as we go and with a three year program with 20 year goals being able to say okay. We have the resources we did the activities we saw the outcome. Outputs allows you to say look. We may not be able to measure this impact or trying to have 22 years the road but we have confidence that if our assumptions are correct. We're making progress in that direction. So this is a logic model that I've put together for the science of security lab late' with help from the pis in that lab. 'let I don't want to read it all to you. But it basically says that there are significant concerns about security and privacy. We lack the foundational theories to address them. We need more scientifically rigorous methods in order to be able to address these problems and we need more people working in this area so in order to address that problem. We've identified some guiding set of heart problems. We're going to focus on. We have a multi-disciplinary multi-institutional team we're working on a roll crafts world-class research facilities and we have a lot of funding infrastructure support to help us do that using all those resources will conduct. Research will provide methodological consulting. An oversight will have workshops and seminars will have community building activities will share data and will have real-time evaluation feedback so if we conduct all those activities we expect to see that resulting in scientifically rigorous peer reviewed publications. We accept expect to see folks. Attending the workshops are participating in our activities. Hopefully that will each new and strengths and connections among researchers in this field and ultimately scientific and methodological process improvements. We hope that that will allow us to have an impact on the larger field of cyber security research. We'll see those principles that we've identified implemented in the design of new systems he enhanced scientific and technical skills within the community knowledge diffusion to the security community in general beyond just the academic community and hopefully that will result in greater national security more fundamental understanding of cyber systems and the security community.

That's poised to address the next challenges and as you can see this is involving not just the researchers with an elaborate. It's the scientific community. The security community is that government concerns. Blows the project scientist students and public in general and so with this framework with these assumptions in place that these various steps will lead one to the next we can identify metrics along the way so we have sort of beat some bean-counting metrics for did we have resources. We need as well as metrics for the activities. So we're going to conduct the outputs we can measure some of the outcomes but as we move along the logic model we're getting into those widely diffused things so we do a lot of effort just sort of document. Here's what we did here's what it produced and we expect that those products will lead to these ultimate impacts and so this sort of time series case study methodology allows you to sort of map your progress along a timeline. So let me just give you some examples of what we're talking about so. I take in the logic model that I just showed you and broken it down one column per slide so these were all the inputs that we had and here are the metrics that we use to measure those so we have the number of researchers and postdocs and graduate students working on the project their collaborators all the different disciplines that they come from we talk about when we look at world-class research facilities NC State as a research one University. That means that we're highly research intensive. University look around you Santino. Campus has a lot of great resources and we're able to use.

We also have support in terms of research infrastructure. We have a program coordinator we have a value ater several others lots of funding. And there's lots of complementary centers and Institute's on campus from la s the Research Triangle Park and then we're able to draw from those resources as well so I forgot to mention each along the bottom here. You'll see sort of a color coding. I mentioned that the lab lat has three objectives and I sort of tried to color code this table to show which objective each input activity output etc is is related to so looking at activities. We've got 13 funded research projects and we can't we report on the progress of those research. Those projects every quarter. We are providing meth illogical consulting and feedback via some methodology seminars. Every week students within that are funded in the lab. Let have the opportunity to present a research pelean or a paper that they're working on and get feedback from the rest of the folks in the lab late' during the first year they had 11 seminars and 18 presentations and based on feedback from the students who presented under percent said that they were able to use the feedback that they received to make improvements to their studies so we know that they're getting that feedback and it's useful to them lots of community building activities so we documented for workshops the community meeting with industry. They formed this new conference hot sauce. Hot science of security so two of those have happened several quarterly p. I meetings have been participated in so those are the the community building activities. We also looked at the publication's to identify collaboration with outside researchers. We saw that they were collaborating with six funded institutions that had co-authors from 13 other institutions outside of this six. And we're also planning to you know when you publish something. That's the result of a collaboration. That's been going on for months or years beforehand.

And so we're planning to do some interviews and faculty to identify new and ongoing collaborations. We are also working to help folks share methodology resources so that we have the hot sauce meeting and so there have been about. Excuse me that they at the house last meeting they had the international research network for the science of security and that's where they share guidelines reading lists they talk about different ways to approach cyber security problems and so we've had about 30 participants in those workshops over the last couple of years and I've been giving them feedback along the way so every time they do some kind of activity we give them. We do a survey about that and give them a report based on how that they can improve okay so outputs if you conduct research you should be publishing it so that year during the 2014-15 academic year they had 20 publications and presentations another 9 under review. But as I mentioned this is the second 30 year award so over the last four years they had 87 publications coming out of this project. We're holding these workshops and these seminars. We're hoping that people are attending them. They are and we're hoping that they are connecting with each other. We can see that not only in their publications. So when you look at the publication's there's 55 co-authors but also in the sense of community that they feel that they report when we asked them about that so ninety-seven percent of the participants who participated in. The hot sauce. Conference felt that it helped build community. Ninety percent met people outside of their working group. We asked industry participants in this conference whether they'd be interested in collaborating with academic researchers seventy-five percent said that they would and I'll get in a minute. But we're planning to do some. CD analysis to look at how these collaborations change over time. We're also looking at based on the methodology feedback that we're providing to students or hoping to see that the methods in our studies become more rigorous and so some folks at the University of Alabama that we're working with are doing an assessment of each of the papers coming out of the lab lit as well as all the papers from the 2015 I Triple E science security and privacy to look at their methodology and compare it to the work that we're doing we're looking at the feedback that the students are receiving as part of the methodology seminars to identify whether or not the studies are becoming more sophisticated.

So if you're doing a study and folks are saying i'm not sure what your research question is. I'm not sure what the goal of this study is. We need to improve. You need to become more advanced as you become more advanced the feedback might say you know. I understand the purpose of the study. Maybe you want to consider this different type of analysis or maybe you want to consider this different type of a metric and that would indicate a more advanced a type of feedback and therefore in a more rigorous type of study. I'm not a computer scientist though. So when we're talking about process technology improvements we really need to go to the source. So we're going to be doing interviews with faculty about what is impacted their work from a technical perspective. Ok so if those are these sort of outputs what is the outcome of those outputs so. I'll talk about this in a minute. I'll come back to that when we and this is where we start saying. Okay we're not going to be able to measure all of this because we're getting past our three year timeline. But we are looking to do some targeted interviews with industry if we're having an impact on the field we're hoping to see that. Come out in the products that are available in the market so we'll be identifying folks that may be benefiting directly from our work follow up with them to find out how they're benefiting from that we're hoping to see enhance scientific and technical human capital and I'll talk about more what that means in a minute but that's basically the skills and the personnel available who've been trained do research in this way.

During the last year there was nine. PhDs and three masters students graduated. That were trained in the program and they've all gone on to take faculty postdoc or industry. Jobs and master students are either working or continuing their research. We'll do some more surveys with students to find out the impact on their skills their knowledge their career trajectory. Okay let's see so knowledge diffusion to the security community if you publish something somebody reads it. They cite it. That's a good way to know that you're having an impact on their work. My understanding of computer science though is that it's a very fast-paced field. Citations are lagging indicator. So we're working on some alternative metrics for that and we're also hoping to see increase in the number of researchers. I can't possibly count the number of people doing research across the world in this area. So that's sort of a long-range diffuse gold so um that was sort of like all over the place description of all the different metrics for using and I want to just sort of focus in on a couple of metrics for each of the goals that we had for this project. So looking at the scholarly outcomes of the research that we had so this is just a reminder that this is a big project with a lot of goals a lot of activities and this is just a metric to access one small part about that effort and so we have both quantitative metrics and qualitative approaches so. I'll talk about the quantitative stuff first so when you're doing evaluation of Science and Technology programs there's some established metrics bibliometrics is one of those ways and this is data from a food science evaluation that. I did and so this is the usual way that I might go about the established sort of way that I would go about. Assessing the impact of somebody's work so the web of science has drawn impact factors general impact factors are based on the average number of citations received by a publication within a two year period and each journal or conference is grouped within a category of similar conferences and journals within the discipline.

And so you can say here's the impact factor of the journals. We are publishing in. Here's the impact factor of journals in that category. And you can see that the journals we're publishing in are almost twice as impactful as average journal in that category so we know that we are publishing high-quality journals not so fast for computer science. The existing bibliometric databases are insufficient. So the web of science is the most commonly used data source but computer science researchers typically publishing conferences and those are severely lacking within the web of science they focus more on journals. They've got a lot of publication venues y se publication venues. I'm talking about journals and conferences that are miscategorized chemistry journals founded as computer science journals and as I mentioned before this is scientists citation based indicator so it's based on the average number of citations to a publication and that in computer science things move really quickly after a couple of years. Things are sort of outdated and so. It's hard to wait that long on a three-year project to see the impact winner and colleagues did an evaluation of web of science and one other bibliometric database and found that a third to two thirds of computer. Science publications are missed in these databases. So we can't use that however the science of security involves not only computer science researchers also involve psychologists statisticians economist so. I wanted to see if maybe it did a little bit better with our work no. I took a random sample of our publications and only twenty-six percent of them were identified in scopus so we need to develop some alternative way of measuring the impact of our scholarly output did look around for some other existing databases.

They're not there so we need to identify some other non citation based metrics for publication impact or potential impact. So we're working on some alternative metrics and if you look at a publication venue there's a couple of sort of numbers you can get from it. We can look at publication selectivity so out of the total number of publications that were submitted. How many were accepted. And you can look at scale so how many publications is a this has been you put out in a year and so we're going to use those two to evaluate the impact of or the potential impact of publication venues. Where our work is appearing. We can also look at co-authorship diversity. So we're trying to do multidisciplinary work so we can look at the different fields in which co-authors are come from. What departments do they come from and we can show that as a as a measure of growth in the science of security community of practice. But the main thing that we're working on is a custom bibliometric for the science of security and that's going to be based on expert evaluation expert evaluation is option using the tenure and review process sort of an established method. It's recommended by the computer research association as a way to evaluate faculty and so what we're going to do is develop a list of what are all the relevant publication venues for our work and then we're going to ask identify who are the appropriate experts in our field outside. The lab can write these publications top tier second tier third tier and then we're going to ask them to actually do that. Ranking and that will allow us to demonstrate what percentage of our publications are appearing in top-tier venues. It will also help us improve by allowing us to target our research to the top tier venues so that's the quantitative work. We're doing on scholarly impact assessment. We also have some qualitative efforts so as I mentioned we're going to be conducting interviews with faculty um we need to understand the impact of their work.

They're the experts we should ask them. So um faculty are already putting together impact summaries and we're going to build on those to help sort of identify what is the impact of the work who might be benefiting from the work that we could actually follow up with. So it's not just a self-report impact we're actually verifying that with outside beneficiaries and it can also help us to identify collaborations that are not always evident on. CBS as I mentioned those are so lagging interview. And so we're working on the interview guides to do those and just to give this is a methodology that we've used for other program evaluations so just to give you an example the type of question we might ask about their work in your opinion as your project produced any significant scientifical scientific and or technical results that field knowledge or research gaps. Please explain and language the lay person can understand and then we asked for validation of that impact have you received a new. Wars or external recognition for your research okay so I'm also. In addition to looking at the impact of their research we also want to look at the impact of these program activities on their methodology and so again a reminder this is just one part of the overall evaluation that we're so we sort of have to two metrics that we're using to assess methodology so students in the prop in the lab let present their research at methodology seminars. They get feedback on their on their presentations. Ninety percent of the students are actually using that feedback to improve their work. So we're sort of hypothesizing that the feedback that they receive well allow them to improve their methodology and then that improvement might be reflected and the type of feedback that they receive over time so if we collect all that feedback we can do what's called. Paula tative coding. I don't know if you guys do that.

But basically it's taking some text and trying to make sense of it using codes so you would take a piece of text and you would identify you know. Can we apply coach this text and then count you know during the first seminar. We saw a lot of feedback about research questions during the second seminar we saw more feedback about analysis and results and so we want to see if the feedback reflects improvement over time. So we're working on that we're also doing and this is actually work. That's being conducted by Jeff Carver at the University of Alabama. He is doing a rubric based assessment of the computer science. That means the computer science security and the lab. What publications so he just submitted a paper reviewing all of the papers in the 2015. I trip police science and security proceedings. Each one of those papers. He went through and said what type of study is. This is an experiment so the model is the case study. And then based on what type of study it is does it include all the elements that we would expect to see in a methodologically rigorous paper based on this type of methodology and so he's just submitted a paper based on that which will say you know. Here's the status of the field. These are all good papers. They've all been accepted in one of the top then using our fields. So what is the standard acceptable level of methodological rigor now based on the various type of studies that are being conducted and then he's working on a review using the same criteria to evaluate the lab lat publications. And so we can say you know. We're on par with the standard of our field. We're pushing the envelope. We need to improve. What's the case okay. So finally the the third goal assessing the community of practice outcomes and this is really about so again one piece of the larger puzzle so I talked before about scientific and technical human capital. What that means is the it's the scientific and technical know-how of people in the field. It's their their social capital.

So who do they know. Who are they able to work with. Who are they able to connect with to get resources and this the more you know and more people you know. That's sort of a more mutually reinforcing network. So you learn more more people want to work with you more. You work with others more you know we. We expect that to have a mutually reinforcing impact on on careers. So we're going to do a couple of metrics to assess the human capital one is creek elem data analysis again. CD is a historical document so it allows me to look at. What were you doing before you started participating in this project. What were you doing while you're participating this project and what happened after that so we can look at. Is there an increase in the number of co-authors there an increase in the diversity of co-authors mr. increase in the diversity in terms of disciplinary as well as institutional co-author diversity this project isn't really focused on commercialization activities however. I think that those might be a spillover effect so we can also look at impacts on patenting and other commercialization activities as well as funding so it's not just about publications so some sample hope so this is the sort of the quantitative accounts from the CD analysis we can also look at collaborations in progress. So have you made visits to the lab of scientists other universities public firms to discuss your project related work. So how many have they visited you if so how often have you actually started collaborating with any of this folksy that you visited or who visited you in terms of career outcomes. We're really looking for for students so we want to know what training opportunities do they. Participate in. This program has a lot of different training opportunities but not everyone participates in everything so we want to understand. Which ones do they participate and does that does which one of those training activities has an impact on their career outcomes so that we can say you know folks who did this this type of training how these outputs and we can pass that along.

So different training modalities different opportunities for collaboration or those were they satisfied with those training opportunities. What impact did it have on their knowledge related to the specific topics of research. There are methodological skills and their skills that working as a member of team and then for students will ask them about their career goals that they're still in school we don't we can't ask them about their careers. We'll also following students that graduate from the program ask them about what impact of their training have on their career trajectory. What kind of positions do they hold. How did the training they receive prepare them for their current positions so comprehensive evaluation. We've got a lot of objectives. And then therefore we've got a lot of metrics we're talking to a lot of different people as data sources and so this is really a case. Study based approach or on-site we're embedded with the researchers. We're really trying to understand a holistic comprehensive picture by using multiple different data sources multiple different metrics putting that all together to create a bigger picture both quantitative and qualitative what is impact of this research and so we're not only trying to measure the impact. We're also trying to help them improve as they go along. And this is really based on some established methodology in the science technology and innovation studies field. But we're customizing it for this this project so a lot of this research is developed in collaboration with the folks in the lab. 'let so I just want to acknowledge their your support in their feedback and questions are chromatic. You thank you any questions so you talked a lot about different kinds of trying publications and it occurred to me that a lot of the computer science things they might be published or providing tool boxes are sliding packages right.

So that'll be part of the faculty interviews so when you're doing this is the science of security so it's not the innovation of security or technology so we're really expecting to see scientific or academic scholarly output more so than the commercial products but we will definitely include that in our in our faculty interviews to ask them about sort of those more tangible the products that they might be developing and when we do the interviews with industry that will definitely be the focus. I have actually google questions and the first one. I she sort of golf data was precisely with what it was just ask is to me and I may have missed it though. It seemed that you never really talked about or accounted for rather the the actual performance of the sciences. Not was how well it actually performs in in maintaining you know the integrity of the cyber network and right so let me go back here. My logic model. Well we can see it here so impact on enhanced security of us cyber systems. It's a long-range goal. We're focusing on. How do we do research on secure on cyber security. How do we understand what are the fundamental principles of secure cyber systems. What are the series methodology that we would use so we're talking about. We hope that that will ultimately be the case but I highly doubt that we will be able to see that kind of impact within a three year time frame. So we'll ask about that in the faculty interviews. What is the impact of your work. Tell us how its impacted cyber security in the US but I don't think it's possible to measure that in a quantitative way because I didn't free your ties from some of the folks that you working with also our algorithm development and thing like that luxury and address these very good. It is very problem right right so and that's been the way that computer science works a lot. You know you. Have you have a problem. You develop a solution to that problem and so the criticism has been that that's sort of a reactive approach.

Somebody hacked something you develop a solution and there's a new problem. We develop a new solution. And it's not. It's you know sort of berry tart reactive and not a broad approach so we're trying to develop fundamental theories that could be sort of influencing the research of others right so so while they do that work that's and that may ultimately be the ultimate goal that may not be the immediate outputs that they're producing related to this project the other question. I is ok so you this some theoretical framework to assess you know to be evaluating the signs security at all. How do you evaluate your evaluation right so we can look at comparison groups so compared to other students in cybersecurity who are or in computer science. Who aren't part of the lab. 'let what does their career trajectory look like. Compared to our students career trajectory we can do the same thing the curriculum vita analysis the bibliometric approach that i had originally proposed where you compare journals that you're publishing into journals and other categories that's a good sort of control group. We're going to have that in this. Because it's so customized but when you're not able to do an experiment have randomization and real control groups. You have to use something like this a logic model to say you know. This is what we think will happen. This is why we think will happen. It will happen and you sort of use measured measured counterfactuals to determine whether or not what you're doing that had the impact of it with something else so one or any other questions all right if not the painter speaker. Thank you appreciate y'all coming out.