All right I think people would be better off listening to the panel than me so I have essentially one observation which is that this panel grew out of the observation that economics is a strongly hierarchical field it’s like the court at The Mikado or maybe this is probably unfair to the Egyptians ancient Egypt it’s it’s it’s structured with departments with journal of economics in India and then vast numbers of folks at various levels but you know in 2008 this process that was supposed to certify things as scientifically sound the best thinking etc you all know you hear this stuff all the time completely failed are nearly so I mean there wasn’t one doctrine that went down it was a whole chain of them and so we thought hmm it might be good if I net looks into what’s the story here can you actually what can you say about the or the the way this structure functions can you actually prove it functions the way you think it does and at some point somebody will want to ask the question so how does stuff how does this actually work to certify things if they go so badly awry what happened and so we commissioned a string of papers than we’re delighted when George and Jim and the other noblest said they’d help out in this so I’m just gonna we’re just gonna do the panel and George is gonna speak first let’s see whether we get some slides we got it I don’t know whether I did that somebody else okay so sins of omission that’s my time okay so there’s a general problem in economics that problem develops out of what I consider a misconception of the field so there’s an almost universal view amongst economists that economic research is about applying scientific method to economic and social problems now I have no beef with that but then economists have further views have further views regarding what scientific method means it is commonly interpreted as meaning the application of simple models and statistical analysis as in physics economists who had wrought adopted this methodology have made excellent contributions so they are to be priced but but it is also true that such exclusionary definition of methodology leads to a dangerous multitasking problem it incentivizes intention attention on problems that are easy to pursue with the exclusionary definition and India incentivizes attention away from problems that are difficult to pursue so let me do a beef summary of what I’ve said so far.
Economists have an almost universal view that simple modeling and statistical analysis are the full domain of economic research as a result as a result there will be economic or social problems that are outside the domain of economic research ok so now let me give you some motivation so in this afternoon’s session I’m going to talk about sins of omission with regard to motivation let me give you a snap preview of that economists have their standard ways to present to present motivation so we make models based on a priori assumptions for example consumers are maximizing utility firms are maximizing profits and then behavioral economics allows some small deviations that but the fact is analysis of many economic problems require a broader view of motivation in this broader view what motivates people are the stories they’re the stories that people are telling themselves at the time they are making their decisions so just think about it that view of motivation is central to sociologists so sociologists think that these stories a much wider and richer range than what’s implicit in economists a priori assumptions so that’s why the core of sociology is ethnography ethnographies empirically uncover they uncover people’s stories they can also uncover the social networking that generates those stories so this afternoon I’m going to give five examples in which such stories play a central role economists restricted range of stories and their consequences is one major example one major example of sins of omission that result from a narrow view of economic research so if you want to see solid evidence that about this and the fact that economists have a narrow view solid evidence that sociologists and economists have different analyses too many economic problems can be seen there’s a wonderful YouTube of an swindlers brilliant sociology one course and is it’s a fun to watch ok and now if you look at that you’ll see that there’s surprisingly little overlap almost none between her treatments and those in the standard economics textbook such as for example Mankiw which is a very good book so economists restricted range of the stories that people tell themselves when they make their decisions is the major reason for this difference so that’s my first example the fact that economists don’t look at this wide range of stories let me give you a second example of sins of omission now you may or may not agree with what I’m gonna say now ok but we can see the Queen of England famously asked as probably all know on a visit to the LSE why economists had not predicted the financial crisis of 2008 so Raghu Rajan has said that would have required detailed knowledge of theory and institutions in the disparate subfields of finance real estate and macroeconomics this also agrees with Ricardo Caballeros analysis which is seemingly de regarding why macroeconomists failed to see the crisis so Cavallaro divides macro models into what he calls core and periphery now he says the core is a DSGE model so this model chooses simple a priori assumptions at many nodes a modelling choice so for example it’s a neoclassical growth model with technology shocks consumption and leisure determined by maximizing intertemporal utility investment on hiring decisions determined by intertemporal profit maximization and rational expectations by consumers and firms so it’s the whole hog so there is in fact and there was in fact considerable macro and finance literature that is peripheral to this core that periphery explores deviations from the core but only one at a time so Caballero explained why that was the case the periphery so here’s this simian.
The periphery is about isolating specific mechanisms therefore it surrounds the sources of these mechanisms with assumptions designed to kill unwanted effects that would pollute the message but but the financial crisis involved interactions that would not be one at a time deviations so a model that incorporated such deviations was therefore outside the range of permissible research now that explains where our theories did not our theories in fact could not predict the crisis even though actually when you go back and read it they’re just fantastically good because you look at each of the individual elements and people have their very good paper in journal of economics in India zones now it’s also true that a statistical approach wouldn’t work so the needed forensic information that could have been available would not be statistical so again it would be outside the range of permissible economic research so implicitly Raghu Rajan in his 2005 Jackson Hole paper in journal of economics in India back word engineers what such evidence would have to be so we should look at civvies what our civvies civvies our financial intermediaries of size and strategic position for which an adverse shock will lead to a fire sale crash fire sale shut crash to remind you is a vicious cycle of force sales of assets that lead to declines in asset prices which trigger further declines in force sales of assets now such danger zones come from managers who take on tail risks and they do so for four reasons first to artificially increase their current reported profits to gamble for resurrection after an adverse shock because they have made miscalculations and because they’re too optimistic so in each of these cases these managers what do they want to do they want to hide their positions so that means that means that the second mode of permissible research statistical analysis will be limited the data is rare because it only involves those tail risks.
Sippy’s but is rarer still it’s rare is steel because the long term Capital Management’s and the casinos and the skilling they need to hide the vulnerabilities of their positions this means that a lesson plist ik v o of theory and a more forensic view of what constitutes evidence is needed to predict where the next crisis will be coming from and this time is different Reinhart and Rogoff suggest that we actually need such analysis but economists have limited incentives to see where how and when such a crisis would come so let me say a few more words the occurrence of sins of omission in economic research as the result of multi-tasking biases should not come as a surprise peeling the onion back one layer further to theoretical papers show that scientists desire for conformity will result in specialization of scientific belief those beliefs include views such as we have been discussing on the appropriate methodology and content of research for the respective field there’s a nice paper in journal of economics in India by Brock and durlauf who has shown that convergence of the leaf occurs with great generality it occurs as long as subsequent generations of scientists shade their beliefs to conform to the beliefs of the previous generation and there’s a recent paper in journal of economics in India by by myself with Pascal Messiah that is modeled conformity more specifically on the biases scientists express in promotions to tenure in the model in the model such promotions and committees favor candidates of the same beliefs as themselves and disfavor those with different beliefs in this system there’s also long-run convergence of belief prosper research methodology then converge as to the following do what I do okay that’s the rigidity of belief regarding what constitutes research in economics as articulated in the mul Itasca model is predicted strongly it’s the equilibrium outcome it’s the equilibrium outcome of the system of promotion and publication in the econ in the academic profession okay so conclusion I’ve given you two examples to get examples that are mainly from my own research interest regarding sins of omission.
I’m sure that everybody here everybody here at each of your tables will have your own examples of sins of emissions of things that somehow you wouldn’t know how to write a paper in journal of economics in India on that topic and yet you knew that that is an important topic so I think this is where I net thank you is he playing a huge role in economics poor form so what is I meant doing i net is expanding the domain of what economic research should be so I think everybody in this room and perhaps myself more than anyone else present is very grateful to earn it and I must say we’re extremely grateful to Rob Johnson for his truly extraordinary personal and selfless dedication to this task so thank you very much thank you okay Jim Heckman is our next speaker so I’m going to talk about some facts that sort of put some flesh empirical flesh on the concerns I think of this entire panel and talk about the way we promote and reward economists and I use the term the curse of the top five because it is wait there should be 12 minutes I only got 10 I’ll keep going okay so what’s and also this is not moving why isn’t it moving could somebody help me advance it oh here okay yes you have to point it this way so really let me just summarize what my main message is.
I mean there there’s something called the top 5 which guides I think the careers of many people here in this room especially the younger scholars the top 5 itself is determined by citation counts and publication is an important gateway to tenure and promotion so it’s an avenue of how the profession evolves and I think understanding that how it involves is what the practices are really important so I want to talk about this practice and what if anything should be done about it the whole idea of evaluating fields on the basis of our scholars on the basis of the journal of economics in Indias in which they publish is a fairly recent development the last 30 years started really in biology and I’ll just let you take a look at these quotations by the way that the entire talk is available if you just write me or write Tom Ferguson I guess there’s a hole for this so the point is is that in this field and this topic the journal of economics in Indias that are considered important journal of economics in Indias are basically those that where there are a lot of other people reading the journal of economics in Indias and other citations are present so the danger is that when a publication is evaluated by the citation factor it’s not really evaluating the article itself but it’s company the company it keeps and so it’s really kind of an odd process if you think about it and it’s exactly as George was saying that you know this produces certain distortions if you give people a reward they’re gonna go in that direction where the reward is and I think in our profession that’s a huge issue so I’m gonna just give you who that what’s the top five are the top five are influential I want to argue but they’re far from the sole source of influence or outlet for creative economics and so here are the top five you haven’t seen them so for those who aren’t directly in academic economics the top five are the are these items here in the left the American Economic Review we kind of metric a general political economy quarterly Journal of economics in India of economics and the review of economic studies and then there are other journal of economics in Indias that are general interest journal of economics in Indias and then a whole series of field journal of economics in India and as we move down this list this is considered further and further from the frontier and further and further from what is high-quality research and so if you look at what the top five is and what the ranking is just in terms of citations you’ll see that the one journal of economics in India that’s the most influential is not even on the list of the top five it’s a journal of economics in India of economic literature and a lot of the journal of economics in India some of these a journal of economics in India Finance is another issue where if we just played the game purely by citation alone we would actually find that the top five is really not the top five in terms of where citations occur but I think the concern about the curse is real and we’ve done a number of discussions which I’ll just summarize number of analyses of tenure decisions and time to tenure at American universities so we’ve surveyed a lot of scholarship a lot of journal of economics in Indias I’ll just summarize a very extensive study so there’s a strong influence of top five publications on tenure outcomes and the rest of the publications off the type five are really playing a fairly weak role not only in time to tenure but the probability of ever getting tenure so this for example here look departments the top five departments ranked by I guess it’s US News and World Report pretty arbitrary but no less staying with that this is the rate that the so called hazard rate the time to tenure and how long it takes for example and what the rate is and what the benefits are republishing one paper in journal of economics in India at just at least one paper in journal of economics in India in the top five and the big black curve on top you can see and top five is much bigger separated from the the dashed curve below and this is a pattern that appears if you look at the other publication ranks the top the tr-85 the so-called general interest journal of economics in Indias and tear publication in those journal of economics in Indias doesn’t really separate who gets tenure who doesn’t at what rate and if you go down the pecking order in terms of departments you still find this true and departments ranked lower than the top five so six to fifteen and you still see a bit of a separation but if you move into the lower rank departments then tier 4 a which is the good field journal of economics in Indias are playing a stronger role but still the top five is very dominant and again this is just kind of saying the obvious that if you publish on the top five you know the probability of getting tenure and the first spell of untenured tenure track employment is much much higher and again that separates out so if you look at the general interest journal of economics in Indias publishing in any of these others is just not going to give you much much of a and and this is certainly true in the top five you ranks if you look at the departments that are second tier if you will sixteen to twenty five then publishing to top five is really what it takes but there really is a hierarchy and the top five the the top five is playing a major role so it there so since I have to move on quickly these are all available the time to tenure plays a huge role the top five plays a huge role in sorting people out but then I erase the question is the top five a good productivity measure and remember the goal itself so I’m not even challenging and I would think you one should challenge if citation is the right measure but the I would argue that even in its own terms the top five is an inadequate measure and I’ll show you some some interesting statistics so the top five is the shaded area this is the distribution of log citations for articles published.
That’s a log scale and you’ll see that the median is pretty high for you know you take e to the fourth you’re getting about 150 150 citations so here are the various top five journal of economics in Indias do you can see and the whole average has clumped together in the shaded area on the left but if you look at the general interest journal of economics in Indias these are articles published in 1996 as cited in 2016 and you can see that even there 20 years ago you can see that places journal of economics in Indias like the economic journal of economics in India and the review of economics and statistics a lot of overlap in some of these other general journal of economics in Indias and if you look at even the field journal of economics in Indias there’s a tremendous amount of overlap about where the citations are now if you go to a more recent measure about articles published 2005 then you can find that actually the economic journal of economics in India and review economic statistics which for many people is sort of falling off the cliff into a non-tenured limbo world or maybe hell the underworld is actually cited just about equally well with many of the top 5 so even by our own internal location citation counts we are actually you know not really being consistent with ourselves so qjq plays a big role now we ask well if we ask what are articles published in the last 10 years by reap ex top 10 authors again.
I’m using a citation account repack what you’ll find is yeah top 5 plays a big role but it’s also the case that a huge number of publications even among those top drank scholars are off the top 5 so by no means is top 5 monopoly we have different ways to measure this and no Nobel laureates and so whether you have to adjust for the volume a eer publishers probably about twice as many articles in a given year as the other journal of economics in Indias so that you have to adjust for that so the articles on the right here the graph on the right is showing that you know the Brookings papers actually are one of the most highly cited journal of economics in Indias and that’s way off the top five by any measure and yet it’s highly influential at least as saying where people are talking about this looking at Nobel laureates and you can do the same thing for Clark Prize winners with a slightly different composition but Brookings papers still plays a very powerful role what I will also want to talk about very briefly is the ranking of journal of economics in Indias there is a kind of an internal inconsistency if we go within fields and we look at scholars who publish from that certain fields and I’ve taken you know development is a field econometrics as a field finance game theory IO and so forth you can see that the categories what you see is the top five journal of economics in Indias are shaded but you can see the leading journal of economics in India in each of these fields typically is not a top five Journal of economics in India where the action occurs is not in the top five Journal of economics in India it’s it they’re important and they’re differentially important across subfields in economics but here I would say there’s a fundamental contradiction here what we’re doing is letting specialists who themselves primarily published in field journal of economics in Indias are deferring to generalist journal of economics in Indias to screen the quality of their colleagues the people who receive honors recognition and tenure in their own profession which is I think event so the top five I guess how much time two minutes okay so the top five accounts for a substantial portion of creative and highly non cited work I will just point out that some of the top five articles of age you look at the repack citation count the most cited article yes it’s review of economic studies so the first five looked pretty good by already the number five Lucas’s paper in journal of economics in India is not a top five Journal of economics in India and if you go down the list you’ll see the Journal of economics in India of econometrics Journal of economics in India of Financial Economics and so forth play a major major role so by no means when we looked at the effective ranking where people are going where they’re looking to in their citation counts that’s and of course the top five ignores any citation to books and with it then the punishes scholars who engaged in long term publication programs that are really trying to do research so here are some very influential books that are just off the top five charts and so we can ask the question about whoops yes the percent of scholars most cited publications so here I’m looking at Nobel laureates and people who are highly cited what fraction of their articles are published in the top five and you can see you know I’m just going down a list of people here George I think you’re there somewhere right no you’re not sorry doesn’t mean to exclude you this was done in haste but you can see it varies a lot so if you look for somebody like Richard Thaler the recent Nobel laureate only about what 30 percent of his papers that he published on the top 5 al Roth 70 percent but it’s not far from the majority of the papers and there really are some extremely influential papers highly cited here Spence I thought he was on the program so I forgot you George sorry but you can see a lot of his papers are actually not in top 5 and they’re highly influential so let me just then make one one observation the professions reluctant the reluctance to distribute gatekeeping responsibility to the non top 5 journal of economics in Indias is inefficient and here’s I’ll take a figure from some work recent work by David card and della Vigna well you’re seeing that the profession is grown over time so what’s happened is the number of submissions is increased that’s the dashed line and the number of acceptances is actually decreased so staying within this top 5 you’re getting a bottleneck and you’re not getting you’re just simply not getting a reflection of what this growing body of work is doing so it’s it’s there’s a lot of work the standards are getting higher we’re becoming more selective and then I finally just say there’s a danger here and the danger is that many of the journal of economics in Indias that are in the top 5 have very low turnover and at variance here this is just a distribution of the average length of tenure breach.
Todd has a policy you have to leave every four years q je the you know you can see there’s a very long tail and other journal of economics in Indias you know jpg and some of the house journal of economics in Indias typically have more what that does is that it actually creates what I would call the potential for incest it creates in the sense I’m not gonna say there’s great bias and so for example take my own University of Chicago you’ll see that if you look at the publication’s in top five by the work affiliation of the author there is an over-representation in the journal of economics in India political economy of Chicago people either Chicago PhDs or Chicago and the same thing may be true for example that at the at the q je you’d find some evidence for example substantial Harvard and MIT playing a very substantial getting a very substantial contribution if you combine Harvard and MIT about thirty three percent of the total publications so that leads to some danger about this in citing that what George was talking about it’s not deliberate bias but there is just a natural sense that we were playing games or playing ball with people in our own thing so let me just summarize this is more of a factual presentation I hope we discuss the long-term implications but I think the top five is undeniably important no doubt it is it’s not internally consistent though in this sense we do have very good predictors of citation counts that aren’t on that like mentioned GL is really the top ranked Journal of economics in India and the economic journal of economics in India and review economic statistics is more or less on par with the top five in terms of citations but the deeper issues are that it basically is not looking at the content of the journal of economics in India and the three excuses faculty from reading papers and making tenure in hiring decisions and as George said it’s really discouraging economist young scholars from taking on creative topics and doing journal of economics in Indias.
I mean you’re George is a living example right your work your your fundamental paper in journal of economics in India on lemons floated around and that when it was first published it was that a journal of economics in India or when it was published it was a journal of economics in India it was pretty low down in the pecking order at that time now it elevated has been elevated partly by the publication of his paper in journal of economics in India so I think though the idea of creativity is really stifled and I think also one thing I’ve noticed in working in China is that many journal of economics in India many scholars working outside the United States want to publish in American journal of economics in Indias they see American topics so I’ll get frequently the topic I can’t work on China even though people are fascinated with us going on in China because I can’t get a q je a JP e and a e ER it has to be something on American data do you get this irony of precious Chinese data being thrown away a lot of problems simply because they don’t think they can get it published and I think there’s this question of centralization which I think it has serious consequences for the health and profession so thank you and sorry for going over time thank you very much our next speaker is Carlota LED maybe Karl Carlo before you one quick comment which as you can see in that list of journal of economics in Indias I think I’m correct that there’s no economic history journal of economics in Indias in that at all you can see the almost hopeless position for a field that almost can certainly benefit by knowing a bit of it says one of the many people if you want to get this paper in journal of economics in India every and you’re on that list I know for sure it just got admitted I was very conscious of putting your name on the list I feel embarrassed that he got but-but-but Fogel published very few of his papers in the so-called top 5 and these are journal of economics in India in Singh with Al Roth and Vernon Smith there were a lot of a lot of so-called marginal people who then later got very famous because their work finally caught on but when they were young scholars they were not they would not have been successful by the top 5 so that’s a huge thanks Thomas attempts for inviting me well in your introduction you started by highlighting how economics is a very hierarchical discipline there has been some research recently on these for example you can measure the hierarchical nature of economics by looking at how many women gets a PhD in economics with respect to other disciplines or you can you can look at the wide range of income distribution within the economics profession that’s another good indicator I think and then this very concentration also emerged when we look at citation so for example more than 40% of citations from top 25 not top 5 journal of economics in Indias points to another topic anomic not this is almost double with respect to what happens for example in sociology or in political science so there is a very very much strong concentration at the top of the economics profession and this has led some people to think about at least to inquire whether we are actually facing and an old boys network or gatekeeping phenomenon as Jim McMahon was actually I will first try to confuse this theory so so for example has been a recent article published on the review of economic studies showing that editors of top journal of economics in Indias which usually come from very very few top economists departments in the US tend to favor the format PhD students or their current or former colleagues both when accepting papers or if we look for example the length of the papers that they accept but I think there is an objection as I just said how can we not how can we be sure that in fact we are not talking about to put it simply the best economists how why couldn’t we for example argue that we are simply observing you know the met the best people simply liking each other’s paper in journal of economics in India and therefore accepting it for publication or citing it when there are authors for example if you look at the distribution of citations this is for Italy but if it’s almost the same everywhere there is a very large share of economies there are almost never cited and very very few ones who get 100 thousands of citations so and they not the best and is it just right that they you know gets this far over very high share of citations well I think it will be interesting and that is one I’m doing in this working paper in journal of economics in India that’s been recently published by net to compare this case is with another one which could be equally credible in my view and that is we are actually talking about people that personally know each other to test this theory I look quite far from the top I took at the case of Italy which has arguably not in the top of most Citation rankings and I do not look at just a few journal of economics in Indias or a few departments I instead I collected data on every single academic economists tenured an Italian University in the last six years and for each pair of others I tried to gather measures of their personal relationships so you may have in my database measures of proximity for example how many paper in journal of economics in India these two economists have written together in the past measures of overall for example how many common institutional affiliations they have as you will know many people have more than one institutional affiliation for example in either it is very common that people tangled in public universities will then teach at private ones or they there they could be attend affiliated to any think-tank for example then I gathered a few measures of similarity of topics this is done with a very sophisticated improvement technique from information science if you’re interested in to the methodological details please please look at the paper in journal of economics in India and the number of journal of economics in Indias in which both economists published and then I also counted how many blog posts or newspaper in journal of economics in India articles commentaries or interviews these people have given in the same time range now there are good reasons I think that to argue that given the highly polarized nature of the media debate in Italy this could be taken as a measure of let’s say similarity in terms of political ideology or political orientation well there are very tricky methodological issues when you look at pairs of others but if we just focus on the impact on the probability that one economic side another one what we observe is for example that having been quoted in the past makes one economist more than five times more likely to cite their former quarters than anybody else in the sample having written in the same journal of economics in Indias make it more than three times more likely to sign to cite another economist and writing on similar topics as more or less the same impact of being affiliated to the same institutions then even riding on the same newspapers or web magazines has a small but statistically significant impact in fact that is also the variable with the higher standard deviation so its impact is even higher than you observing this graph but what I think is interesting is looking at the joint impact of all these variables together I mean if we are talking about people that are affiliated to the same institutions they may have worked together they published in the same journal of economics in Indias and often on very common topics well in Italy you would call them friends maybe in anglo-saxon country you could talk about colleagues or acquaintances but there I think that there are good reasons to think that citations measures something more and different than just sheer scientific quality well but then there is another contour argument and one could raise these are all idiosyncrasies of every any kind of pair of people that you could pick in my sample is it not likely that for example this in this video sync receipts just cancel each other in the aggregate so I did another an exercise and I aggregated all citations that every economist Italian economist again received in my sample from anybody else in the sample and I expressed this as a function of a number of personal characteristics so for example being woman is associated with a lower average number of citations this is very typical in most scientific disciplines I must say it’s not special of economics and then I also introduced a number of centrality measures in all those networks that I was looking at before so again the media outlet networks the wardship network affiliations journal of economics in Indias and topics and what I find interesting is that often many of these measures turn out to to significantly increase the number of predicted citations that an economists receive at any year but in particular what turns out to be statistically significant is closeness centrality now this is interesting because you know you could define the centrality of a person in a network in many ways for example you could count how many people does this person know and this doesn’t turn out to be so fundamentally driving the number of citations instead what I found to be really important is how close you on average to everybody else in the network and we measure the distance by how many links how many people are in between the minimum number of people in between any two economies so again we observe that being very close to everybody else is fundamental in the calc ownership network is very important in terms of affiliations very important in terms of journal of economics in Indias and somewhat important again in terms of political ideology as well what is not important as being central in the Topix Network and this is also if you want relatively interesting because there is in many disciplines ample evidence that it is it does not really pay to be an eclectic writer or for example interdisciplinary studies on average usually get a lower number of citations than very special topics or very specialized orders so on the whole you know in the bibliometric literature everybody has always say that citations at the same time measure both scientific quality and somebody and something else the typical argument is that if the discipline works well then well you could argue that being in a socially central position in network terms or in whatever terms you want is a reward for intellectual achievement instead I think what the Italian data shows but one could argue that I mean it’s just an example it’s a relevant whole country for a case study is that being there is not associated to many other things that do not have anything to do with intellectual achievement such as one’s gender someone’s ideological positioning and several other variables so possibly being in Italian and a little bit too warm and I should not be talking about friends but at least one could argue that many people cite basically their colleagues and of content says thank you Thank You Carlo our next speaker is dr. Marcella Corson thank you and Italy again as you can see another city hmm it’s quite crucial I mean already on the presentation before me as stressed important so keeping diversity in our profession and of course I will also speak about Cyril yzma in that sense and why Italy well of course Conroy’s already introduced our country but actually Italy probably you know it has a long history of pluralism and the terra docks economics so you wouldn’t expect I think something what I’m going to show you now concerning the actual Italy and the natural Italian Academy today so because actually Italy has become an example of how bubble matrix can become really a tool for censorship somehow or in any case a tool for discriminating against diversity and certainly at our Doc’s economics we have something run by the state that is very similar to what actually Jan Akkerman has introduced you before this is not just a matter of having an interview when discussing subjectivity about what is better what is wrong it’s just that the state itself has decided that you are right and that you can go on in the academic profession if you respect some specific requirements so how is done all this is done with this specific new system is called the billeted scientific an optional ESN as you can see that is a system of evaluation for single searchers but also for governing academically as of a promotion of to associate and full professorship altogether there is a committee of five members randomly selected there are no interviews and no interaction with the candidates everything is done just mainly by the biometrics economics actually it’s not a it’s not defined by law as a bit of matrix sector but as a matter of fact it is this means of course and I will show you in a second some results there is there is really risk losing pluralism they started pluralism of our economic Academy and of course in this risk there is also risk of AM elongation of AM elongation between women and man you know if you were there yesterday you know already that this is moment one main point of my on my analysis women and men tend to be different in terms of interests a scientific interest certainly they are different types of performance in terms of productivity and so on there are a lot of studies about that there’s no point for me to to go in detail about what it what I mean but let me just give you some figures concerning women economists in Italy definitely gender matters there is an under-representation of women in university it is a standard figure at European level you certainly know the she figures published by the European Commission only thirty six point five percent of tenure academics in Italy are women for all fields of course in economics we are just thirty point three so and forty four and despite the fact that we have forty two point five percent are women in PhDs there is a gender imbalance in the academic position because less than eighteen percent two full professors are women and more than forty five percent our assistant professors there is a persistent vertical segregation with a classical pyramid for women so a large base is made by researchers then of course in the middle the associate professor at the top the full professors but if you look at the man career so you find exactly the opposite situation so all women are concentrated in the top so in the full professorship then you have a smaller number of course among associate professors and then a very small number among researchers so absolutely the opposite kind of your our key structure and then just was playing very briefly girls I don’t have the time to go in detail but please read the paper in journal of economics in India and come to us this work is actually carried out with a body poet and Juliet Zakia was there so we are ready to give you all the information that you might require so what we have done is to build up a proper database just taking into account all the candidates today today a essay and as I said in the last two rounds so Mitch means 2012 and 2013 for each of them we of course are taken into account all the information available on the website run by the National Committee for evaluation or research so called amber but we also explored a little bit more introducing other information coming from the main data set secondly Tan and Rebecca or Google Scholar as well I don’t have really the time to go into details so but just to give you an idea what are the difference in terms of women and men here you find the candidates for full professorship and the candidates for associate professorship you can see the percentage of candidates who were actually qualified we doesn’t mean that they found really a position it means that they can apply for a position and this is another big problem by the way so 48.3% qualified for full professors 47.7% qualified for full professor but if you look at the women candidates the percentage is absolutely lower 31.6 for full professors and 39.8 for associate professors as I said before but I don’t have the time this is just to tell you what we have done we have built an enormous data set mainly thanks to Julian Zakia did a great job on that and we could actually explore more in detail their gender identity their research interest the publication there are bits and he had my presentation I will give you some links for those of you are interested in exploring a bit more the work that we have done but just to give you again a quick quick view of what happened with sa with a sin yes and basically I told you before there are no interview there are there is no interaction with the candidates the five members of the committee’s just receive the publication’s and more or less read them of course you cannot really check whether they read them or not but they have to provide of course a proper report about each candidate and basically they have to look for for the respective Trish or three criteria the first criteria concerns the number of monographs the second criteria the second criterion concerns the number of journal of economics in India articles and book chapters a third criteria the target Irian concerns the number of articles in the so called alias journal of economics in Indias and of course for the our a-list journal of economics in India supplies most of what Professor Eggman has already said about the top five of those the top five are there for sure but there are many more but in any case there are there isn’t any Italian journal of economics in India in that list Oh better there is only one published by Springer all the other historical Italian journal of economics in India and are speaking about journal of economics in Indias in which published Nobel our rates like Modigliani and so on are not included in that list just to be clear but anyway these are the three criteria as you can see women have performed better the man in that sense seventy-seven percent of women had succeeded in passing the threshold for the second criterion and forty seven point seven forty seven forty eight forty four seventy four point seventh in the third criteria but still if you look at the final result and this is something that we had done with specific statistical tools if I’m not if I’m wrong please Kellogg or enemy this is actually done with the censured probate introducing of course all the characteristics that we were able to pull a pool a pool in our data set introducing also does information that I told you before from google books Google Scholar Rebecca Ecole eat and so on then if you test and I finish with this just to give you an idea if you test the probability of getting or being qualified in Ellison being a woman of beer man if you try to qualify or if you look at the qualification for associate professors that is actually the line on the top there are not large differences between women and men candidates but if you look at the u-shaped curves instead that apply to full professorship being a woman means that you are absolutely less favored in getting the qualification and I’ll leave you with this thank you so much this our publication thank you very much Jakob capella is our next speaker before he starts though one quick comment which is when you see numbers or statistics and pictographs like those last ones of marchello one hope we entertain a dinette is that if folks think about suing in these different countries the laws are different different institutions that some of this evidence might become relevant it’s a little hard to imagine that the whole profession will reform but conceivably it’s happened that lawsuits have made institutions much more careful many thanks for having me here and we’ll talk about citation metrics and research quality in economics yeah I just wanted to say that and it would really be glad if you could put on my presentation instead of Kevin’s also I would be honored to present your paper in journal of economics in India all right will do Kevin first since there’s obviously a problem with the order of the presentation great so my name is Kevin Young and this is a paper in journal of economics in India that represents joint work with Lassa Henrickson and Len Seabrook both of the Copenhagen Business School this is part of a larger eye net funded project called embedding groupthink the name of this paper in journal of economics in India is fathers of neoliberalism so what we’re trying to understand in this paper in journal of economics in India has to do with a historical question of the neoliberal ascent whereby essentially a minority tribe within the economics profession eventually ascended to popularize a very very important sort of economic doctor in the world that’s a story that’s been told many times question of whether or not that neoliberal ascent happened is not in question here but the how question which has been examined in a variety of scholarship looking at example the importance of a variety of right-wing think tanks or the importance of the elements of the business community mobilizing the 70s that story’s been told what we want to investigate in this paper in journal of economics in India is the professional practices and performances of economists over a multi-decade period leading up to 1985 when we feel the neoliberal ascent has eventually become consolidated so our starting points is our that neoliberalism had to replace an existing ideology the battle over the dominant economic paradigm had to be struggled for socially and professionally and this struggle of course took many different forms but a very important part of that struggle had to do with what was going on within the profession of economics what was going on within economics profession was absolutely critical because the kinds of act the kinds of ideas that other actors used whether or not their later neoliberal politicians or think of how the business community mobilized neoliberal discourse later all of these ideas had to have popularity more importantly they had had scientific Authority and they had to have a cadre of experts sort of waiting in the wings that could be called upon especially when it came time to make adjustments to the ideology as was as it was developing so our approach in this study is to examine economists level of performance and aspects of their behavior their professional behavior historically over this period that we we analyze and we want to know very specifically did neo liberals outperform their non neoliberal peers in terms of publications grants or perhaps in terms of their career trajectories in terms of moving into more government positions or other differences in their career trajectories importantly for us this is a story that we want to examine across generations and so here’s our analytical set up for this for this paper in journal of economics in India we find what we call descendants or children of neoliberal fathers so there we have Milton Friedman and George Stigler there think of their family tree think of all of the all the people that descend from them all the PhD students they had over the years as that as the sort of transition was happening in the 60s and 70s we want to find attributes of the professional standing both of these fall and of their children of their PhD students and we want to compare that to it oops okay we want to compare that to a comparison group that that is similarly matched in terms of institutional prestige in other words the reproductive platform has to be matched from Chicago to something else and how we did this is as follows we gathered 13 what we call neoliberal fathers there’s some nice pictures of them some of which are historical some of which are more recent so this is not you know every professor of the University of Chicago during this period but these are people that are really pushing the pre-market line and we’re involved in this in this intellectual movement are what we call Charles River fathers our economists of similar esteem and standing in the 50s 60s and 70s that also made important contributions but we’re from Harvard and MIT the Charles River bisects the two institutions so we have 13 neoliberal fathers and 13 Charles River fathers what we do then is try to find their descendants so this is a network representation of the forward path in a genealogical Network of all of Milton Friedman’s students over the years so this is his students and then his students students going on you can see that the effect can be you know quite important if you if you supervised a lot of students but we actually found that the repec data had a number of biases in it sort of present as biases as it were with respect to generating more elaborate family trees for more neoliberal oriented oriented economists so what we did is we backtracked and we took our 26 fathers 13 from the neoliberal group 13 from the Charles River group and we worked really hard to find different children different first-generation children of these of these fathers we looked at a variety of different sources from fetch rifts and obituaries we contacted University departments we utilized some information from oral histories as well as we visited a number of archives to you know do things like look up reference letters for students to get a sense of of how these inter generational ties were working and we did a number of things to try to this is described in the paper.
I’m gonna go over this quickly I’m gonna show you some graphs so you believe me that we kind of balance the data carefully and and gather it a good number of students now so we have basically two pools of people okay and then we start looking in the paper in journal of economics in India at at performance metrics so this is the mean citation rate the historical citation rate so not citations now but citations that were flowing in at the time if you look at the upper right-hand panel here we’re comparing the first generation students of the neoliberal fathers to the first generation students of Charles River father’s folks of a more Keynesian orientation that were critical of the idea that free markets would emancipate humanity and as you can see there’s not a big difference over this over this period with a potential exception in the late 1970s on the upper right-hand panel whereby the neoliberal children had some advantage the big difference occurs among the father’s themselves in other words they were not able to carry this advantage over into the next generation in an interesting way starting in the 1970s there’s a big big sort of citation ascendance of the neoliberal of a neoliberal group this plot these series of plots are actually more important for us because these don’t measure performance but measure behavior this is the in citation rate right so this is the the degree to which you cite within your own tribe okay so if you look at the upper right-hand panel and even the lower left-hand panel the difference between students citing within their own tribe is not is not very great so neoliberal children and Charles River children are sort of citing their their generation at about the same rate the big difference is again with respect to the father’s so the the both the left-hand panels here are are are the rate at which students cite their fathers okay so there’s a there’s a sort of paternalism effect or kind of ode to the father effect it’s much much stronger in the neoliberal group than the Charles River group they’re citing back to their fathers at a much higher rate about three times the rate as their Charles River peers at the time we also looked at funding or unable to find any real a clear distinct patterns in terms of the simple funding rate how often you it you publish something that was associated with funding across the the three institutions that we studied we also looked at the composition of funding now there are very important think tanks and foundations and specific government agencies that played important roles but in terms of an aggregate picture the the composition of funding is not is not different in terms of the sources we also looked at mean grant funding in terms of just the amount of money you could pull in from the NSF which is the one really systematic database we were able to get and gather and the Chicago folks were not advantaged in some sort of environmental selection effect here in terms of acquiring more funding nor does the effect take off over time as we as we observed it we also looked at career paths so this is an alluvial flow flow plot that just looks at the trajectory of these students out of these institutions and into different kinds of careers you know you know maybe it’s possible that more sort of neoliberal children went into careers and think tanks and international organizations we did not find this to be the case and we looked at this in a variety of different ways we looked at different government positions people had and we also looked at high status and low status academia were unable to find a consistent pattern that we describe in the data now this is this is a slide.
I’ve sort of been waiting for and I’m really excited excited about to the to the students in this in this room were now graduate students at the LSE Eliot Jerry and Elizabeth Sullivan hassan both contributed significantly to this network what we did was we took all the publication’s of all of these fathers and children from 1960 and 1980 inclusive we took the acknowledgement sections right so this would be you know Kevin Young thanks Jim Heckman for that for the useful comments on the paper in journal of economics in India kind of thing and this generated a large and elaborate Network which visually is almost impossible to interpret but thankfully there are Network analytical methods that we can use and by two methods that measure levels of social solidarity as these networks evolve over time both transitivity and reciprocity is much higher for the neoliberal group then than the other groups no matter how we break it down so reciprocity is really simple right it’s like I I thank you you thank me back maybe in a later publication we’re helping each other transitivity is a little different it’s sort of like I I could know someone else sort of through you it’s more of a small world thing being being picked up there but these are measures of social solidarity not of behavior not of performance that suggests that the neoliberal group was very solid heuristic over over the period of their ascent so to conclude the neoliberal ascent we argue has an important professional performance angle when we compare these two groups in terms of historic okay in terms of historic citations there wasn’t a clear pattern except for the father’s in terms of external funding there absolutely was was no clear pattern in terms of career paths people didn’t take career trajectories that were very off the mark from their comparatively matched peers but in terms of in group cohesion we argue that social norms of reciprocity and what we call in the paper in journal of economics in India insurgent solidarity played a really important role in the neoliberal ascent that’s it and and now Jacob develop thank you now okay many thanks for having me here I will talk about citation metrics and research quality in economics I will try to address three issues first I want to introduce to use and research quality and reputational stratification in economics second I want to ask how well can we really assess the quality of individual contributions and third I want to illuminate which signals play a role when it comes to the determination of publication decisions and citation behavior let’s start with the two perspectives on citation data that is something that was already subtly present in the other presentations why do we need two perspectives the simple reason is that citation data is typically used in two different ways either positively or normatively a positive approach would be to employ citation data to analyze the properties of sides of the scientific conversation or the development of scientific fields and this is typically dubbed as cognitive scientist the second approach to dealing with citation data today is much more popular this the idea to employ citation data to evaluate and measure the scientific performance of authors editors departments Eternals and so on so we have a kind of evaluative Center metrics and would argue this is a really important difference because for instance if you look at a specific group of self-referencing people you could say that from one perspective this must be a group of highly talented geniuses but from the other it might look like a citation cartel and in what follows I’ll try to explore this a bit with reference to the big five the topic we already had so what you see in this chart is the share of references in the top five that originated from the very same set of shonas that originated from the top five and we believe this is an interesting measure of discursive concentration and we find that more than one quarter of references in the top five 27.5% also originate from the top five.
Now what does it tell us to get a better cross economics here is really unique or exceptional evader this is something typical in science we made a series of interdisciplinary comparisons we compare the results from economics which those we get from studying sociology political science and sociology and what we find is that these other fields have a much lower degree of discursive concentration whereas the value for the big five was 27.5% the value drops for sociology and political science to about 12% and for sociology to about 8% now when I present this to mainstream colleagues the response often get is that this is has nothing to do with groupthink or discursive concentration this is simply because the sociologists political scientists and psychologists have very lazy editors and these lazy editors are weak in identifying high-quality contributions whereas editors in economics have put in a lot of effort to select the papers with the highest merit so we have again two different interpretations one interpretation would be the cognitive one saying that what we find is that economic discourse at the top is very concentrated and self-contained and reflects the general astark hierarchies in the discipline where is the alternative you’d ever let a evaluate if you would say that economic simply as a small set of highly regarded journal of economics in Indias which are able to select those papers with the highest merits and hence high quality research is so easy to locate and it’s no wonder it’s it is intensely reference so taking this claim we were interested in does this really work could it work does it work this argument based on the thesis that we select the papers with the highest merit to assess this argument we built a small agent based model in this model you have a population of authors every also produces a manuscript then there is a self-assessment of the manuscript quality and which a submission decision is based referee gets the paper in journal of economics in India also makes an assessment and then is the publication decision and we have two variants of the model one in which we have an ideal world and everything runs perfectly objective okay every estimate of qualities every submission decision is objective this is the one version of the model and then we introduced a slight degree of error in assessment in the self assessment in referee assessment and the modest dose of ambition when it comes to the submission decision and what we find is that in our ideal world we really have a perfect ordering of 12 with regard to quality we can perfectly order channels whereas in the slightly less ideal world we have sweet optional sending out and the rest is a volatile mess we can find no clear pattern now we can use this model to ask for the fate of top papers so what happens really to the best papers and you know ideal world we find that everything is as it should be so the purple line is the rejection rate and then the x-axis you find manuscript quality and we find that bad papers are rejected and good papers are receptive okay everything is as it should be however you know less ideal well we find that the rejection rate has a be modal distribution where repair papers are rejected but also rather good papers are rejected with a very high probability and this is due to a bottle net effect that arises due to the increase in ambition when it comes to submission behavior so now what happens what is the fate of the top papers in the ideal world when we look at the 20% of papers with the highest quality we find that all those papers get published and most of them in highly reputable journal of economics in Indias okay however in a less ideal world we find that of those papers with the highest quality more than 40% remain unpublished at the end of the day due to the bottleneck effect already observed and only about 55 percent get published okay so the implication of agent based model is basically that the argumented only intrinsic quality of paper in journal of economics in India matters is employs it really does not even work in a next to ideal world where there are no fat snow fields nor social networks just a slight error in the estimation of quality and the bit of ambition when it comes to making decisions about where to submit a paper in journal of economics in India so when we when we find that all the intrinsic quality matters is implausible we correspondingly find in the empirical data some indications that signaling plays a decisive role and relevant signals the surf is a proxy for manuscript quality have already been discussed here we have been informed that signaling social capital is important so past professional context impact publication process we have found evidence that signalling competence is important we found some indications that the top 5 are cited because of the prominent position in the discipline will not necessarily because of the high quality of all the contributions in there we find something that James Heckman has already shown limited PhD origin and affiliation matter so origin is important and we also find that membership is important so where you belong to marys and in this case this refers to paradigmatic member membership so I wanted to tell you about something about signalling origin and have collected numbers on the those institutions which which are which have the greatest degree of representations and top journal of economics in Indias in these members are basically very similar to the numbers James Heckman showed so what we find is that for the three top channels a are Fujian GP more than the half of authors have the PhD from one of the top 10 institutions and more than one-third of others have the affiliation with this top 10 institutions let me go down a second example for a signal I would argue that signalling membership is important.
There is the long-standing finding that within the economics literature there is a certain prejudice against heterodox economic contributions so paradigmatic origin is also a signal that is relevant when it comes to publication decisions and citation behavior and the clearest evidence is simply if the comparison tation behavior of heterodox and mainstream economics journal of economics in Indias where we find that this is on the right bar that heterodox journal of economics in Indias are quite open for mainstream input but the mainstream genres are almost closed for heterodox input so we find in our large-scale data set that only one 7% of references in the database come from heterodox runners in the mainstream there is more than 98% come from the mainstream itself so what we argue is that paradigmatic long and paradigmatic orientation is a kind of relevant selection mechanism again this is the question what does happen here is their form of institutional closure going on or is it simply the bad quality of heterodox economic research and one way to assess this is to assess this question rise heterodoxy neglected is it prejudice or that research is to look at those papers in the mainstream that actually cite heterodox work so what you have here is in the x-axis the number of heterodox articles referenced within mainstream papers and on the other axis you have the change in citation impact relative conditional on the shona so and what we find here is that basically mainstream research gets more interesting then it also incorporates heterodox references and i would argue that these amounts at the end of the day to a self-contradiction of the evaluative perspectives if really heterodoxy is rejected because it’s so bad then there is the question why the introduction of heterodox sources in mainstream works makes the letter more interesting so to wrap up i’ve introduced two views and research quality and citation data is it all about quality or is it about social structure.
I’ve asked can be selected paper in journal of economics in India purely on merits we have seen this does not work even an ideal setting however we found the perverse effect that many high quality contributions are ignored and finally we find some indications that this kind of ignorance is not random there is social capital origin signaling competence and kerlick mathematic membership plays a key role many thanks for your.