User:Apovolot/Expert peer review
This page contains my proposal re Expert Peer Review process for the Science related articles. This page also contains ensuing discussion re that proposal. This page is created as a copy of the "Wikipedia's Expert Peer Review process (or lack of such) for Science related articles" section in my regular talk page. Some fragments relevant to this discussion is also posted on http://en.wikipedia.org/wiki/User_talk:Jimbo_Wales Apovolot (talk) 23:54, 3 November 2008 (UTC)
Dear Wikipedians,
It appears to me that the full scope of Wikipedia's Review process for Science related articles only arises "by force" when the article is slated for deletion (is nomination for deletion constitutes more or less "by chance" random event ? - please correct/clarify my guess on the "randomness" of that deletion nomination process ...). Further, it appears that this Wikipedia's Review (which seems to be only arises in full scale during the argument for deletion ) is conducted in overly-democratic fashion where every "editor", irregardless of his qualifications in the area of the discussion, is allowed to have a voice.
Of course I understand that the "consensus" requirement is built-in into this process to alleviate possible "ignorance" of the majority "nay say-ers" ... but since in such review there is no mandatory participation of the person with sufficient "expert" credentials, thus (in my humble view) the "consensus" requirement is not fully working as designed (to preclude from the ignorant judgment outcome) and the review "quality" is less than to be desired (also note that such review process doesn't conform with the scientific publication review requirements due to lack of appropriate qualifications among the reviewing peers ...).
Would it be beneficial (for the Science articles only) to introduce the list of "participating" "qualified" reviewers, of which at least one should be involved into each review ?
Also it appears that there are possibly 3 areas of qualifying the Wikipedia's Science related article (each qualification area would require its own list): a) Scientific content b) Editorial (style, format. etc.) appearance c) Legal conformance
Could we start the discussion on that ?
Thanks, Best Regards, Alexander R. Povolotsky Apovolot (talk) 14:02, 25 October 2008 (UTC)
Please note that in my proposal I am only suggesting to use expert editors for science related articles. I could retreat further and suggest that such review (by expert editors) could be used just as an an optional feature (again for science related articles) either requested by the author of the article (for the purpose nomination of the article to become attested as attaining the "scientific publication" quality ) or by admin/editors during the deletion discussion, when they feel that the participation of experts is needed to decide on the issue. Apovolot (talk) 14:52, 26 October 2008 (UTC)
On another hand (optionally) Wikipedia may consider to extend its charter (from being just an encyclopedia) and create/keep the section for the "original scientific research" postings - being clear marked as experimental and NOT fully reliable info ... That would allow Wikipedia to perform / stand as a "catalyst for change and innovation". Apovolot (talk) 16:32, 26 October 2008 (UTC)
It sounds to me like you got off on the wrong foot. There are general guidelines as to content and style that are generally understood and are generally followed (although there are sometimes mistakes, errors). You need to get more familiar with the customs, the procedures, and the style that is commonly employed at WP, before trying to get into deep discussions about process. linas (talk) 15:24, 25 October 2008 (UTC)
Linas - your reply appears (to me) to be too formal ... I did a small research before posting ... I am not talking re general Wikipedia customs, the procedures, and the style which guide Wikipedia articles in general - I am only talking about science related articles and pondering whether those could/should achieve the quality of "scientific publication". I didn't find anything on that - sorry. Apovolot (talk) 15:42, 25 October 2008 (UTC)
See Wikipedia:Expert editors, a proposal that was rejected, and Citizendium. EdJohnston (talk) 19:28, 25 October 2008 (UTC)
You could look at GA review and FA review. Also the various wiki-projects in the science field. In terms of "scientific publication" the answer is yes and no. WP compares well in many respects with a lot of text books, but is not the same as, for example, a research paper. Rich Farmbrough, 01:20 26 October 2008 (UTC).
The basis of wikipedia is verifiability, not truth. GA and FA articles are in no way guaranteed to be up to par to what experts in the field consider what is actually reality. For example, the first sentence of Natural Selection is not about natural selection, but about Evolution by means of natural selection. But heck, there are a bunch of editors who smack around with citations, satisfying the verifiability, but at the same time promoting the wrong idea. So, I, as an expert editor, gave up. Wikipedia is inherently anti-expert. -- Kim van der Linde at venus 19:19, 26 October 2008 (UTC)
It was, I think, Winston Churchill who said "Democracy is the worst form of government except for all those other forms that have been tried from time to time". You could say the same about Wikipedia's policy of "verifiability, not truth". The idea of "expert editors" or "expert reviewers" sounds plausible at first, but sooner or later it would degenerate into "experts" endlessly squabbling over content. "Experts" are just as obsessive, quarrelsome, petty-minded, intransigent and incoherent as the next person - and probably more so when they feel their intellectual alpha status is being challenged. Wikipedia needs an objective benchmark to resolve content disputes, and "verifiability, not truth" is a pragmatic and successful approach. Gandalf61 (talk) 09:16, 27 October 2008 (UTC)
Thank you for proving my point. -- Kim van der Linde at venus 10:10, 27 October 2008 (UTC)
Ah yes - I should have included "sarcastic" in my list of qualities. Gandalf61 (talk) 11:02, 27 October 2008 (UTC)
- I would agree with Gandalf61 viewpoint. Scientific articles are subject to review like any other article, but it can't be left to experts to decide what articles meet Wikipedia policies and guidelines, since they are indistinguishable from the rest of us. The process of review is not as random as you might think: since every article can be reviewed at any time by anyone, statistically speaking every article will be peer reviewed at some point. If it does not comply with Wikipedia policies and guidelines, then part or all or the article can be challenge by anyone - which seems quite democratic to me. I like the Churchill quote, but even though democracy can be chaotic, it does enable issues to be debated and resolved by the people who are participating in building Wikipeida.
--Gavin Collins (talk) 11:17, 27 October 2008 (UTC)
Yes, most articles are peer-reviewed, but most of the time, experts just let it be, because they recognize that fighting the windmills of democratic ignorance is not something you can win. -- Kim van der Linde at venus 14:11, 27 October 2008 (UTC)
Dear Gavin.collins and Gandalf61- thank you kindly for the expressed opinion (which is appreciated) - but why do you want to have the review process be left to chances and statistical probability (no matter how high this probability could be) ? Why do you suggest that the experts are indistinguishable from the rest of us ? In my view the experts should be willing (at least temporarily during the review) verifiably reveal their true identity and credentials, which should be at least at the PhD level in the corresponding area of science (or higher). I am not myself in high admiration of those "obsessive, quarrelsome, petty-minded, intransigent and incoherent" high priests of science but do believe that using them for TRUE (in its scientific meaning) peer (expert) review is still better than existent highly anarchic form of review. Democracy is OK, anarchy and the rule of ignorance is not. Cheers, Apovolot (talk) 01:22, 28 October 2008 (UTC)
Despite the name, Peer Review on Wikipedia is not any sort of expert review, such as is done for publication in professional journals. Peer review here is to improve articles - do they make sense, do they conform to the Manual of Style, how can they be improved to meet the criteria at WP:GAN and WP:FAC, and in some cases do they properly present the topic? The problem at Peer review here is how to get enough reviewers to do even that - for the past 8 months I have done several hundred PRs and have just cut back from burnout. While I like the idea of expert review on Wikipedia in theory, the practical details would be daunting if not impossible. How would the experts be found and recruited? How would their identities and expertise be verified? For example, if I signed myself up as Stephen Hawking, how would that identity be disproven or verified (I'm not him, by the way). My experience is that people who know about topics (have expertise) tend to raise these issues at WP:FAC, not peer review. Ruhrfisch ><>°° 00:47, 28 October 2008 (UTC)
P.S. I do not claim to be an expert in anything here on Wikipedia. I often wrote in peer reviews things like "I am not someone who normally writes about X, but ...". Take care, Ruhrfisch ><>°° 00:51, 28 October 2008 (UTC)
Dear Ruhrfisch - I am not denying that the practical details are to be addressed but I am optimistic re expert review feasibility and therefore do believe this should be attempted at least as an optional (by the special request) feature. In my view the experts should be willing (at least temporarily during the review) verifiably reveal their true identity and credentials, which should be at least at the PhD level in the corresponding area of science (or higher). Cheers, Apovolot (talk) 01:09, 28 October 2008 (UTC)
Hi Apovolot. I think that the questions concerning verifying credentials are overblown. Most researchers have an easily accessible webpage or something. The criteria could simply be at least one academic publication and institutional affiliation. The larger issue with what your propose is that it'd be extremely difficult to find anyone willing to throw the weight of their credentials behind an article. It sounds so time-consuming that I would never agree to endorse a mathematics article, but I will happily contribute to articles in areas with which I am familiar.
There is an honest problem with certain science articles. The answer to these problems is rigorous application of the standard wikipedia policies (e.g. NPOV). Our only mandate as wikipedia editors is to simply quote published materials without bias, interpretation, or synthesis. If some of us happen to be experts, then maybe we can do this more efficiently (or perhaps better with respect to exposition). I would say, however, than a good number of editors do this remarkably well despite their lack of formal qualifications. Editing wikipedia articles would be a fantastic exercise for undergraduates. From an educational point of view, I would love it for my students to come here and contribute. I also think they'd do a damn good job. Rapid response to disruptive editors and dogmatic application of the usual policies is entirely sufficient. I also think that the various subject projects (e.g.Mathematics Project) are very helpful and should be bolstered in some way. In my experience, things have a tendency to go terribly wrong when someone starts claiming that their expertise is worth more than a citation. Typically, these cases involve some sub-par researcher or practitioner who lacks an institutional affiliation. I would endorse a WP:EXPERTADVICE page that outlined the wikipedia policies and goals for researchers in a way that enticed them to edit here in an appropriate fashion.
Maybe if we were all sensible, then we would think "Editor x is an expert in this area, I ought to ask him what he thinks of the article." Perhaps a well-maintained list of expert editors with institutional affiliation would facilitate this sort of highly informal review process. I don't think anyone would object to a well-maintained list of highly-qualified researchers with institutional affiliation (but then again, everyone seems to object to something). Put such a thing in your userspace or something. On the other hand, if we're following the guidelines exactly, then experts are really only useful to help improve exposition. No article should be so specialized that verification requires an active researcher. Also, we would have some clear problems if the list had any formal meaning. shotwell (talk) 07:13, 28 October 2008 (UTC)
I am not going to go on one side or the other but I have had experience with this situation in the past (if you are reading this, wait until the end because it seems at the beginning that I am taking sides). First of all I would like to point out that:
a) It is very likely that someone may not have any qualifications but still be an expert in his field. Being an expert in your field, and having qualifications are two different things. You could take Gauss for instance; he was quite intelligent in his earlier years but would not have had any credentials then. Therefore, it is quite likely that there are some very knowledgeable people participating in science discussions (or maths) who do not have any qualifications as such.
What does it mean to have a PhD? If you hear that someone has a PhD in some field you would probably instantly think that he/she is intelligent (which is true). But it is possible for someone to know a field at the PhD level and be able to make new discoveries without having any qualifications. You are assuming that everyone who is a serious editor of Wikipedia has a PhD which (as far as I know) is not the case. As I mentioned earlier, age is a factor to take into account.
Yet another point: which 'expert' would 'waste' their time editing Wikipedia. There are some people out there who like to share their knowledge but currently, in my experience, quite a good deal of people don't think of Wikipedia that highly. In fact, some people (especially experts) get the opinion that Wikipedia is 'bad' because anyone can edit (we both know that this is not the case though). So for know, you probably have to work with the current Wikipedians and I am quite certain that most of them will be knowledgeable (even if they have not won a noble prize!).
b) However, I agree with you that experts should participate in discussions but then there is the 'identity problem'. I claim to be a 'topology expert' but I may not be. I may be a very bad student in topology who thought the name is suitable. If people judged me from my username, it would have very bad consequences on Wikipedia. You can't really say that someone is an expert unless you look at their edits. I admit to have made some mistakes (mathematical) in the past but I should not be afraid to edit Wikipedia. My point is that you can't judge people by little things on Wikipedia and nor can you judge them by their username. Unless you see what they write on talk pages and judge their ability by their edits on a whole, there should be no rule against them participating in peer reviews.
I have seen many mathematics peer reviews in the past and very often, the article is rejected to become a feature article because 'it is not explained well enough'. Many experts may find that the article is well-expalined if not 'over explained' but then Wikpedia is for anyone. If someone wants to learn a subject, he/she should readily be able to understand a certain concept in Wikipedia (knowing the required prerequisites of course). So my point is even if a peer review is rejected, there may be good reason for it (even if it does not seem so at first glance).
I partially agree with you with regards to references. References are always important, whether it is in a publication or to verify a claim made by a Wikipedian. Even if someone is an expert in his field, he quite often would have some misconceptions. Suppose I can put it this way: I claim that a regular Lindelof space is not necessarily normal (a misconception that Urysohn had!) but it is already well known that this is not the case in the mathematical community. If I go and put this on Wikipedia and claim to be an expert, I am spreading misconceptions over a larger community. References to published material are always good because they have been judged by a wider community.
It would be good to include experts in discussions and perhaps Wikipedians should notify experts when a peer review takes place. But then no-one really knows how many experts are actually there on Wikipedia in a particular field. In fact, I learn about a new mathematician in Wikpedia every day! But then the same problem arises: how can you tell whether someone is really an expert. When I find out about a new Wikipedian, I do not immediately judge their ability depending on whether they are annonymous or not.
My point is that later on, more people will be involved in Wikipedia and Wikipedia will expand. There are several math articles that really deserve notability but currently do not have much content; for instance topological group. I have many times had disputes with people regarding the correctness of content in a math article; sometimes I am right and sometimes I am wrong. This is probably the same with experts, and really, you can't always treat an expert's word as 'God'. Perhaps the best thing to do is: in the presence of a peer review which seems to be going 'out of hand' notify someone who is in the field (not necessarily having a PhD) and check his opinion. But even if he seems against many of the opinions presented in the peer review, you probably should not completely accept his word. I think in general most of the participants in the peer review would be talking some sense. It is unlikely that all of them are incorrect. Hopefully, as more experts come into Wikipedia, there will be less disputes but disputes are a part of editing and they will probably always be there. Topology Expert (talk) 09:42, 29 October 2008 (UTC)
User:Topology Expert also posted in "What about this" section of this talk page (see below) the following:
Dear Apovolot,
I suggest the following criteria for someone to be an expert editor:
1. They should have edited at least 100 distinct pages in their field
2. They should have made at least 500 edits in their field
3. They should have participated in many peer reviews (how many?)
4. They should have edited pages which are considered at the 'centre' of their field. For instance, in mathematics, someone would be considered an 'expert' editor if he has contributed significantly to pages such as: Lie group, Topological group, Group (mathematics), Fibre bundle etc... (concepts which are crucial to mathematics).
5. They should have also edited at least 50 concepts which are quite specialized in a field. For instance Moore space (in topology), Essential range (in measure theory), Projective space (not so specialized) (in projective geometry), Banach space (in functional analysis), Cohomology (algebraic topology) etc...
6. They also should have collaborated with a good deal of other Wikipedians
Checking whether they have a PhD is not of much use as I mentioned earlier (and is not a fair criterion to check whether they are really an expert). Please tell me what you think.
I agree with above proposal but I still feel that additionally expert reviewers should be able to demonstrate some level of the verifiable accomplishment / recognition in the domain of professional science (everything mentioned above relates to amateur Wikipedia activity ... ). I do understand current Wikipedia concept that in order to produce good Wikipedia science article, one does not need to be a professional scientist ... - that is fine with me ... But I propose to have (at least optionally) ability to review/qualify such article by professional scientists. Those professional scientists should be willing not to hide under the veil of anonymity. BTW, I do not see any reason why the anonymity of editors on Wikipedia is considered to be a "good thing". Above is my general opinion, so please don't take my statement personally. There is obviously a choice given for everyone in Wikipedia either to act "in open" or to hide behind meaningless assumed pseudonym and I accept this situation. Apovolot (talk) 16:24, 2 November 2008 (UTC)
I haven't read the whole thread above, but just my 2c: while I think we need as good reviewers (for GA, PR and FA) as we can, setting up a formal "expert" label is difficult. Also, it is not really needed, in my experience. The "expertise" of editors is best measured in their edits (which gives a fairly good way to judge people's abilities), both for practical reasons and for the reasons that people may well be "experts" in their fields but are not (yet) expert writers on WP. The main problem I see for all these review processes is the lack of reviewers at all (not specifically expert editors). Jakob.scholbach (talk) 11:56, 2 November 2008 (UTC) I am sorry to have been so long replying with comments on your ideas: I knew at once I didn't like them, but it has taken me time to set my thoughts in order about why.
I think you are trying to move WP away from what it has deliberately decide to be, towards something more like a peer-reviewed journal. There are on-line encyclopedias which are more like that - Citizendium, Veropedia, probably others - but (though I don't know much about them) I don't think they are making much impact. Your suggested departure from the policy of no original research would take us even further away from our guiding principles.
One of WP's fundamental principles is "verifiability, not truth". That way, the problem of deciding what is truth is outsourced to the editors of the publications we use as sources. Our problem is reduced to deciding what are reliable sources. If we had our own experts claiming to decide what was truth we would be in an entirely different game, and the sort of arguments we have now would be as nothing compared to those that would arise when two schools of experts each had their own vision of The TruthTM.
I can also see hideous mountains of bureaucracy arising in the question of who selects the "experts" and validates their qualifications, and in how their superior status is to work in practice.
So far as your more limited idea of advisory panels goes, we already have something like that on a voluntary basis - the WikiProjects, where people interested in a subject "watch" the project's talk page, so that (to give an example) when in the course of your AfD we needed opinions on the status of OEIS, I only needed to post at Talk:Wikiproject Mathematics and within a short time several people who knew about it commented.
Sorry to be negative, but I think your suggestion would dilute the qualities that (for better or worse) make Wikipedia what it is. Now I will go back and see what others have said.
Regards, JohnCD (talk) 22:13, 4 November 2008 (UTC)
- Having gone back and read the others, I see Nsk92 has put my ideas much better. JohnCD (talk) 22:21, 4 November 2008 (UTC)