How to Identify a "Good" Scientific Journal
I would love to look a lot of this up on my own, but am unsure about what sources can be trusted. I know you talked about how scientists are not created equal, but as an average person without the background to fully understand the primary sources or the ability to synthesize a consensus without reading meta-analyses, where can I go for reliable information?
This listener brings up a great point. If you follow any of the conversations on the Skeptalk email discussion list, you've probably heard us banter this back and forth. One guy says "Hey, leprechauns are real, here's an article in a peer reviewed scientific journal that says so," and then someone else replies "No they're not, because there haven't been any such articles in my peer reviewed scientific journals." Almost any debate can degrade into "My peer reviewed scientific journal is better than yours."
Now, a really satisfying answer to this question would be "Here, go to http://www.legitimate-scientific-journals.com, and you can see at a glance if your source is a credible one." Surely there must be some register like that, right? I will dash your hopes with a simple answer: No. There is no such thing as an authoritative list of reputable scientific journals. There can't be. And the reason is that word "authoritative". Who is qualified to be the authority? No one is. No one must be. The moment that any one group is anointed with the ability to declare a source to be legitimate or not, is the moment that we lose objectivity and impartiality....
As long as we're throwing around the word reputable, I might as well give the somewhat disappointing answer to the listener's question, and tell where you can find a reliable journal. Scientific journals achieve their status only through long histories and good reputations. To be broadly accepted within the mainstream scientific community, a journal must have established a long history of responsible reporting, good quality articles detailing well performed research, and exhaustive peer review. Long standing reputation among the scientists who matter the most. If you're not one of those scientists, it can be difficult to know which research is good, which editors and referees are good, and which journals have a long history of publishing them in good standing. For this reputation to have any meaning, it must stand on its own and not be supported by appearing on some simple list. Unfortunately, listener, you just have to know; but I will give you a starting point in a moment.
While it is essential that good journals be peer reviewed, you should be aware that almost every publication hoping for prominence describes itself as peer reviewed. When you hear someone defend their source by stating that it's peer reviewed, be skeptical. By itself, that's meaningless. Think back to our old example of the guy writing a UFO newsletter in his basement who has a couple of his UFOlogist buddies endorse his writing. Suddenly he's "peer reviewed". This is not the type of peer review that carries any meaning within the mainstream scientific community, since the peers have a clear agenda and have not established long histories of scientific acumen by the legitimate community at large. This is an extreme example, but it does accurately portray a lot of what's out there. When you don't know anything else about a journal, the fact that it calls itself peer reviewed cannot, must not, be allowed to carry any weight.
What about identifying which scientific journals are reliable? Since we're not all scientists in the chosen field with the education and experience to know which are the most reputable publications in our field, we need some kind of list. But, as we've discussed, lists are bad things when they come from a source with an agenda. So we turn again to our source with no agenda, Wikipedia. Search Wikipedia for "List of Scientific Journals" and you'll find that they have a page listing a few hundred reputable journals in most scientific fields. Generally, this is an excellent list. The fact that it comes from Wikipedia, and is constantly being revised for objectivity and quality by experts in each field, is its strongest recommendation. Some fields are not listed, and most subsections are partial. You can drill down to find more. But beware: the further you drill down, the broader the quality control becomes, and the more journals of lower repute are included, and more non-scientific fields are listed. If you use this list to gauge the reliability of a source that you see referenced, you are in as good of hands as are available to the inexperienced journeyman; but you must use the list wisely. Stay at the top level, or as close to it as you can. With each click that you drill down, reputation for the listed journals is generally lower.
Again, this recommendation will no doubt be criticized, and the criticism is generally valid. But I maintain that for the average guy off the street, this is the best way to gain a "good enough" grasp of a journal's quality, and to find good research that truly represents the current scientific consensus.