Part 2 - How the F*ck Do Research Journals Work? Rankings
Unpacking the Status Games Behind Academic Publishing
Every year millions of studies are published in various journals around the globe. This is supposed to be a fount of new knowledge from the best and brightest.
But that is a façade.
The replication crisis in over half of studies in multiple fields and reports of a Nobel prize awarded to a researcher with faked data, tells us that the system isn’t working as advertised. Part 1 of this series covers what academic journals are, the history of how this all got started, and a rough outline of the process of academic knowledge production.
In part 2 we are going into more depth on journal ranking systems and the status game in science.
In Dune, the spice must flow. In Academica, studies are the coin of the realm. This resource is largely controlled by publishers who are raking in billions with a heady profit margin of 40%, much of it at taxpayer expense. This system is held in balance, in part by ranking systems that purport to effectively sort the juicy new insights from the latest pig in lipstick fronted by a PhD desperate to get tenure.
Papers, Papers Everywhere
With the enormous churn in research, sorting the wheat from the chaff is a huge job and an important one if results matter. While there are competing ideas on how to sort all this information, most of them come back to quantifying citations from other work.
Think of how Google determines page ranking by how long a webpage has been up and how many other websites link back to it, and you begin to get the idea.
And much like Google page rankings, Academic journal rankings are wide open to being gamed.
One of the ways ranking is determined is the Hirsch or h-index. This metric attached to the scholar (though it has recently been adapted to journals as well), provides a number value based on both productivity and citations. Even without knowing the specific math behind it, it is obvious that a researcher listed as a collaborator in more papers, and cited in more studies will have a higher rank.
This list is even public, stacking up scientists like sports stars, relevant data is highlighted like the goals and assists on hockey cards.
Looking at the numbers, it is hard not to wonder how these scientists have time for that much work and sleep too. Putting two and two together, these would be the folks that high-ranked journals would want to publish since journals gain their rank from citations too.
Journal ranking unlike the scholar h-index is determined by how often their studies are cited by subsequent work over a two-year post-publication period.
Various ranking schemes use different methods in how they tally citations. Impact factor is a popular rubric that measures the average number of citations. It can be found on The Web of Science and is itself paywalled. Another ranking kept by SCImago is freely available and attempts to account for the importance of journals that citations originate from.
While these methods make for great ways to jockey for status, look at all the Harvard scientists highly ranked on the h-index, by results all these methods are doing a lousy job of winnowing out bad work and elevating what’s ground-breaking.
Even where the top journals aren’t obviously corrupt, the incentives for a circular citation setup, call their credibility into question.
To get a better feel for how this plays out, and how the ranking systems affect the choices people make, let’s follow 2 hypothetical young scholars moving studies through the publication process.
I’m Just a Study Looking To Get Published
The Charmed Study Life
So, you are a young PhD, looking to make a name for yourself. You got your grant money, and the study is complete all you need to do is publish. You are studying at a well-known university, with several other established professors who have high h-index rankings and you get on well with them all.
Based on your field of study, your work is eligible for the top-ranked journals. Since this is your first big study, publishing in a high-ranking journal would turbo-charge your career. It would also make your university, and fellow professors look good for you to do well, especially if you cited them in your work.
Your mentor in the department is very well-known, has published several times in the highest-ranked journal in your category, and has a good relationship with their staff. He has also signed on as a collaborator to your study. Being a collaborator will show as an additional published study toward his h-index.
You write your draft.
There is a little tension in what order the names of your collaborators will appear on your study, but it gets worked out.
You cite your dissertation, several of your rock star mentor’s studies, and some from other professors in your department, giving a bump to everyone’s h-index.
Your paper goes to peer review. It’s accepted, but you have to add some citations to other scientist’s work. You think some of the citations may be to the work of the scholar doing the peer review. You wonder about this, but you need to get the study out asap so you can move on to the next one and grow that h-index score.
You get notice that everything looks great, and the journal will even publish your study open access, which will make it easier for others to find and cite worldwide–another big boost to your h-index, all you need to do is pay $5,000.
You pay the $5,000, the study comes out and your career is off to a good start. If you fudged any data, it will probably be a long while before anyone even notices.
The Hard Knock Science Life
You are a young PhD, at a small college. Your field is not flashy, but it is very important with some implications for business depending on what you find. There is room for ground-breaking discoveries, and you have some novel ideas. A few people in the field really believe in your work, but you didn’t go to an Ivy League school, and if you want to get tenure, you have got to get some publishing done as quickly as possible no matter what.
Your field of study doesn’t have top-ranked journals. You get several solicitations to pay large amounts to get your work published in journals that look sketchy to you. You made that mistake with your last study, so you avoid that trap this time.
You don’t get along with one of the professors in your department, in fact, you think one of them has even acted to delay your publication, to prevent you from advancing before their own study is published.
You send out your study to several journals. You wait. And wait. And wait. Eventually, you get rejection notices from all of them.
You try several more. Again you wait. This time one agrees to publish your paper with some edits. It is a lower-ranked journal which is frustrating because your research standards are impeccable and your findings while not definitive, are very intriguing. You don’t know exactly where this could go, but you do know your take is novel and looks very promising.
You get the edits and some of them are extensive. They include adding citations. You add a few of them.
After even more waiting you are now getting published, but you don’t have the funding for the open access fee, so you only get a few citations, though they are glowing. You rush on to the next study in the hope that this one might be different and that you can get enough published to get tenure.
While these dramatizations don’t begin to cover all the twists and turns on the academic life path, hopefully they illustrate the problems in the incentives in academic publishing. As with any other ranking system that is not based on something directly observable, like the number of goals scored, people will find ways to make the system work for them. Unchecked this can be to the detriment of all, as we are seeing.
Further Reading
Publish or perish: The myth and reality of academic publishing
Help Keep This Conversation Going!
Share this post on social media–it costs nothing but helps a lot.
Want more perks? Subscribe to get full access to the article archive.
Pledge to get video and chatroom access starting in November 2024.
Support from readers like you keeps this project alive!
Diogenes in Exile is reader-supported. If you find value in this work, please consider becoming a pledging/paid subscriber, donating to my GiveSendgo, or buying Thought Criminal merch. I’m putting everything on the line to bring this to you because I think it is just that important, but if you can, I need your help to keep this mission alive.
Already a Premium subscriber? Share your thoughts in the chat room.
About
Diogenes in Exile began after I returned to grad school to pursue a Clinical Mental Health Counseling master’s degree at the University of Tennessee. What I encountered, however, was a program deeply entrenched in Critical Theories ideology. During my time there, I experienced significant resistance, particularly for my Buddhist practice, which was labeled as invalidating to other identities. After careful reflection, I chose to leave the program, believing the curriculum being taught would ultimately harm clients and lead to unethical practices in the field.
Since then, I’ve dedicated myself to investigating, writing, and speaking out about the troubling direction of psychology, higher education, and other institutions that seem to have lost their way. When I’m not working on these issues, you’ll find me in the garden, creating art, walking my dog, or guiding my kids toward adulthood.
You can also find my work at Minding the Campus