Wikipedia:Deferred changes/Request for comment 2016

Oppose (Bots)

Discussion (Bots)

Whatever is decided here, I think it is important to make sure that edits deferred by a bot with a neural network/machine learning setup (i.e. CluebotNG) which are later reviewed by a human are automatically added to Cluebot's dataset. I don't know how technically difficult this is to accomplish, but it seems worthwhile if it is reasonably doable.

I'd also caution whoever first implements this to go very slowly at first. Creating a massive backlog of edits that will never be reviewed is not the goal. Tazerdadog (talk) 08:49, 16 October 2016 (UTC)[reply]

• Assuming that a bot like ClueBotNG would continue to operate as it does now (reverting edits that meet its vandalism scores, or however it works) and only defer edits it is less sure about? Ivanvector (^Talk/_Edits) 17:56, 17 October 2016 (UTC)[reply]
  • @Ivanvector: Yes, that's the plan. Cenarium (talk) 21:54, 17 October 2016 (UTC)[reply]

Should we allow ORES to defer edits passively and actively? Thresholds would be determined by consensus initially, and administrators may set higher thresholds in case of backlogs.

Support (ORES)

1. As long as the thresholds can be specified onwiki. ORES is still in beta and the false positive ratio is quite high for the current threshold, see this for the kind of edits it picks up. We would need a much higher threshold. Cenarium (talk) 09:35, 14 October 2016 (UTC)[reply]
2. I support deferring edits as it prevents the visibility of likely vandalism. It doesn't stop helpful edits. Chris Troutman (talk) 23:11, 15 October 2016 (UTC)[reply]
3. weak support as not sure on accuracy - however as per comments in discussion below it's probably just about getting the correct settings/levels. So support with a careful monitored extended trial period, and a way it can be shutdown/paused if causing issues. Cheers KylieTastic (talk) 22:13, 20 October 2016 (UTC)[reply]
4. Support per Halfak's comment in the discussion section and the above. --TerraCodes ^{(talk to me)} 22:34, 20 October 2016 (UTC)[reply]
5. Weak support because it is useful, but misses certain cases that are more likely to be vandalism. I'll have to try more ORES, but this would be a weak support from me based on what I've seen so far. epicgenius - (talk) 21:10, 21 October 2016 (UTC)[reply]
6. Quite honestly, I'm not sure why this is even being debated. ORES's accuracy can be adjusted; heck, ORES's API can be used by flagged bots. This doesn't mandate use; it only allows use – and provides a native on-wiki queue instead of manual review of automated reverts or ad-hoc IRC scoring solutions, like we have now. Kevin (aka L235 · t · c) 00:07, 24 October 2016 (UTC)[reply]
7. Support, only at a very high threshold. See comments below. Kaldari (talk) 02:54, 26 October 2016 (UTC)[reply]
8. Provided that the threshold is higher. Esquivalience (talk) 00:59, 3 November 2016 (UTC)[reply]
9. Support per Halfak (WMF)'s explanation below. — Gestrid (talk) 19:30, 5 November 2016 (UTC)[reply]
10. Support but with others' concerns taken into account.
    Talk • Work 20:11, 5 November 2016 (UTC)[reply
    ]
11. Support. Systems like ORES can be calibrated to different levels of sensitivity. If it's overly aggressive, it can simply be set to a different threshold. Ozob (talk) 00:51, 6 November 2016 (UTC)[reply]
12. Support, if ORES scoring is set to be sufficiently selective enough to not clog the queue. -- AntiCompositeNumber (Leave a message) 20:06, 6 November 2016 (UTC)[reply]

Oppose (ORES)

Until the vandalism model of ORES improves in accuracy. Currently, it is pretty inaccurate, with a high false positive rate even on its "High" setting. Also, I believe that the model doesn't factor in linguistic features, which is fatal to its accuracy. I believe that until more diffs are included into the machine learning model (perhaps the ones in CBNG), we should hold off on including ORES. — Esquivalience (talk) 01:45, 15 October 2016 (UTC)[reply]

The high setting actually flags more edits than the low setting, so will have more false positives. It is high in sensitivity but relatively low in specificity. The setting for deferral would be kept very high in specificity (to avoid false positives) and would have a relatively low sensitivity. Cenarium (talk) 13:10, 20 October 2016 (UTC)[reply]

Well said. Thank you. --Halfak (WMF) (talk) 20:27, 20 October 2016 (UTC)[reply]

ORES is still under active development and is not yet as accurate as bots like ClueBot NG. Deferring edits based on ORES would be premature, IMO. Kaldari (talk) 22:34, 15 October 2016 (UTC)[reply]

If we set our threshold at the level of ClueBot, we would be just as, if not more, accurate. --Halfak (WMF) (talk) 18:12, 19 October 2016 (UTC)[reply]

@Halfak (WMF): I didn't think about that. What threshold would you propose in order to be "at the level of ClueBot"? Kaldari (talk) 19:42, 21 October 2016 (UTC)[reply]

The ORES service reports a threshold for minimizing false-positives in it's test statistics. E.g. https://ores.wikimedia.org/v2/scores/enwiki/damaging?model_info shows "threshold": 0.959 for recall_at_fpr(max_fpr=0.1). If we wanted to run this as a trial, I'd start there and adjust based on what we learn. --Halfak (WMF) (talk) 15:01, 24 October 2016 (UTC)[reply]

@Halfak (WMF): Thanks for the info. I switched my comments to a support. Kaldari (talk) 02:54, 26 October 2016 (UTC)[reply]

ORES isn't quite accurate in my view for active deferral. It might be selective enough for passive, but that would probably fill the queue unnecessarily. AntiCompositeNumber (Leave a message) 14:15, 16 October 2016‎ (UTC)[reply]

Changing to Support AntiCompositeNumber (Leave a message) 20:06, 6 November 2016 (UTC)[reply]

1. Oppose per points raised above, this may create too much of an unnecessary workload. —
   talk | _contribs) 00:11, 23 October 2016 (UTC)[reply
   ]

   Until ORES is better at predicting vandalism, I wouldn't trust it with something like this quite yet. It has the potential to flood Pending Changes with way too many false positives until its been more finely tuned. — Gestrid (talk) 15:42, 25 October 2016 (UTC)[reply]

   Changing !vote. — Gestrid (talk) 19:27, 5 November 2016 (UTC)[reply]

2. Oppose because ORES is still at "release status: beta" according to its page on Meta. After the beta period is over, I would be able to change my position here. - tucoxn\^talk 13:18, 11 November 2016 (UTC)[reply]

   Also, this sub-section doesn't seem to be counting correctly. - tucoxn\^talk 13:20, 11 November 2016 (UTC)[reply]

   Fixed numbering -- AntiCompositeNumber (Leave a message) 14:01, 11 November 2016 (UTC)[reply]

Discussion (ORES)

There seems to be come confusion about m:ORES' accuracy level. ClueBot NG only reverts a small amount of vandalism because it's thresholds are set extremely strictly. Thresholds are set far less strictly in the ORES Review Tool because it is intended to benefit from human review and the goal in patrolling is to get all of the vandalism -- including the majority of damaging edits that ClueBot NG misses. The nice thing about ORES is that it allows external developers the possibility of tuning the threshold to their needs. --Halfak (WMF) (talk) 20:12, 19 October 2016 (UTC)[reply]

I don't understand what ORES is. What does it do? How would it help compared to regular recent changes or m:RTRC? epicgenius - (talk) 12:43, 21 October 2016 (UTC)[reply]

Hi epicgenius. See m:ORES for an overview. ORES is a machine classification service. It can flag edits that are likely damaging/vandalism for review. The m:ORES review tool surfaces predictions on Special:RecentChanges and a few other places. The review tool is installed as a beta feature on English Wikipedia. You can enable it in your your preferences. The ORES service, however, provides scores that can be used on all sorts of interesting ways (e.g. User:DataflowBot/output/Popular_low_quality_articles_(id-2). Quality control is just one of the use-cases of the system. --Halfak (WMF) (talk) 14:52, 21 October 2016 (UTC)[reply]

@Halfak (WMF): Thank you for the explanation. So, are the classifications automatic from the beginning, or do humans classify the edits in order for the machine to provide suggestions? epicgenius - (talk) 15:42, 21 October 2016 (UTC)[reply]

epicgenius, see m:ORES/What and Wikipedia:Labels/Edit quality. If you want more substantial discussion, see this presentation I gave about the system at the Berkman Center.--Halfak (WMF) (talk) 15:49, 21 October 2016 (UTC)[reply]

Thanks. Anyway, I'll be sure to try it out. I'll respond with a support or oppose later. epicgenius - (talk) 15:50, 21 October 2016 (UTC)[reply]

Given that there is some dispute about whether ORES is sufficiently specific for this task, with the counter-argument that it can be tuned to decrease positive results, would it be possible for the people behind ORES to provide a specific list of example edits which would be matched (e.g. from some contiguous sample of 10000 edits on enwiki)? ⁓ Hello71 15:35, 25 October 2016 (UTC)[reply]

Actually, now that I think about it, given that the edit filter proposal seems overwhelmingly popular, would it not be possible to feed ORES scores into AbuseFilter so that EFMs could tune it based also on other criteria (e.g. user is not confirmed or whatever)? ⁓ Hello71 15:38, 25 October 2016 (UTC)[reply]

Maybe not such a great idea, since it was pointed out that ORES scoring is slow. Maybe it would be worth it for this to split edit filters into immediate and queued types based on the actions taken, but that's a whole different can of worms. ⁓ Hello71 15:54, 25 October 2016 (UTC)[reply]

• How many edits do you envisage this adding to the queue? If this is going to create a situation like IP edits on de-wiki, where the queue becomes so unmanageably long that the edits never get reviewed, then all this will do is add another unmanageable queue to a project which already hasn't got enough hands on the pump to cope with the existing queue at
  Iridescent 10:12, 14 October 2016 (UTC)[reply
  ]
  • Such a situation should not be allowed to develop. We should start slowly, and passively at first.
    1. For an edit filter, we start with passive defer, so the revision displayed is still the latest revision, it's merely added to the review queue (but note that we can filter out passive defers). If this creates a backlog, we make the criteria more stringent so it doesn't pick as many edits, then we can make it active.
    2. For ClueBot NG, we start also with passive defer, use a false positive rate slightly lower than the one used for rollback, then check if this creates a backlog, if not we go active at this level and try a slightly lower level for passive defer, check if it creates a backlog, and so on. (Similar for ORES)
    • There are enough editors who deal with vandalism on this project. For various reasons, anti-vandalism work is more 'popular' than NPP and lots of other areas of the project. It won't take people from those other areas, because there are already enough people doing anti-vandalism work. This is just a way to make this work more efficient, by guiding these people toward the edits that are most likely to be vandalism or otherwise damaging (and prevent their visibility to the public). Cenarium (talk) 11:18, 14 October 2016 (UTC)[reply]
• What kind of message do people receive when their edit is deferred? I suspect that their response (quitting in disgust vs. patiently waiting vs. merrily keep editing) partly depends on the response text. Jo-Jo Eumerus (talk, contributions) 15:51, 14 October 2016 (UTC)[reply]
  • For passive defers, I don't think the user needs to be notified since the edit is merely added to a queue. For active defers, we may send an echo notification like "An automated computer program has requested review of your edit on <page>." plus the log summary of the defer. The notification could link to WP:Deferred changes. Cenarium (talk) 20:17, 14 October 2016 (UTC)[reply]
• How feasible is it to actively defer edits that create a new page i.e. replace the publicly available page text with something like "This newly created page has been flagged by an automated algorithm for human review before being made publicly available."? Such a functionality would be useful against spam, attack pages and copyright violations. MER-C 05:02, 15 October 2016 (UTC)[reply]
  • Replacing the publicly available text would be difficult. Though userspace is now noindexed by default, and noindexing of new pages in article space is coming right now. Cenarium (talk) 13:34, 15 October 2016 (UTC)[reply]
• What's the real need for this? Do we not have enough anti-vandalism tools? I'm concerned that this would make the newcomer's Wiki editing experience even more unfriendly than it already is. SteveStrummer (talk) 02:28, 17 October 2016 (UTC)[reply]
  • We don't have tools that act before vandalism occurs, except the edit filter prevent action and the spam blacklist, but these are too strong so can only be used on the most extreme kind of vandalism/spam to avoid false positives. This provides a new way to deal with edits identified as damaging before they can be seen by readers. Tthere is also plenty of vandalism that gets identified by edit filters but get missed by existing tools for days. As for being unfriendly to newbies, this concern is present in active defer but can be mitigated by watching for false positives and using friendly notifications. Cenarium (talk) 15:42, 17 October 2016 (UTC)[reply]
  • Given we have reasonable thresholds for active defers, I can see this reducing casual vandalism attempts, thus reducing the need for regular editors to deal with vandals. The problem this solves to a point is vandalism that becomes publicly viewable, that, in turn, gives the casual vandal a little "high". Without this high, these folks might well see that the Wikipedia isn't a place where they can get their kicks.
    Talk • Work 20:17, 5 November 2016 (UTC)[reply
    ]
    • Agreed. Even when we revert them, their edit is still visible to everyone (barring any administrative action) in the page history. — Gestrid (talk) 20:23, 5 November 2016 (UTC)[reply]
• Doubts about putting deferred edits in with the Pending changes list. PC currently includes any edit to individually selected articles; the PC patrollers are looking for not only vandalism but BLP violations and copyvios, and have volunteered and been granted the permission to do that job. No-one else who currently patrols for vandalism anywhere but has not sought and gained the PC-reviewer flag could patrol the deferred edits, even though anyone at all can currently patrol recent changes homing in on an edit filter tag or group of tags. No, a new queue would be needed for the deferred edits, and new criteria would have to be agreed to review that queue. Even then, I wonder if the proposer and supporters have sufficiently considered the knock-on effects: maybe there are enough editors who deal with vandalism but is it expected that many of them will move over to deferred edits from STiki, Huggle and other processes, making these processes ineffective or redundant?: Noyster (talk), 10:04, 17 October 2016 (UTC)[reply]
  • I don't see why they should be treated differently than any other pending edit, BLP violations and copyvios should also be watched for, as well as spam, etc. PC reviewers are chosen specifically for their ability to review this kind of edits, and deferred changes can be seen as a temporary pending changes protection (in fact, it's how it's implemented), so I don't see the need for another queue or usergroup. As for a knock-on effect, rollbacking an edit with Stiki or Huggle would remove it from the queue so the processes are complementary. Deferred changes is more of a secondary tool, useful for edits that were missed by the primary recent changes patrolling tools. Cenarium (talk) 15:42, 17 October 2016 (UTC)[reply]
• What is the standard for accepting a deferred edit (especially active defer)? Is it the same as pending changes? In my view, it should be an even lower standard. If we wouldn't have reverted that edit on sight before deferred changes existed, then it should be accepted. I don't want this to turn into a way to reduce the voice of IP editors. agtx 16:05, 19 October 2016 (UTC)[reply]
  • It's a legitimate concern, it would have to be discussed when rewriting WP:Reviewing pending changes for deferred changes. The standard would be de facto lower since there would be no need to "uphold the reason given for protection". Cenarium (talk) 12:48, 20 October 2016 (UTC)[reply]
• This is going to have some serious performance considerations. ClueBot NG takes about 5 seconds to score an edit^[1]. ORES takes 1.5 seconds. Abuse filter takes long enough that its application is split into a job queue. If we were to defer changes at the time of save, we'd need to delay the save operation substantially. --Halfak (WMF) (talk) 20:34, 19 October 2016 (UTC)[reply]
  • This doesn't have to be done at the time of save, I've added it in a deferred update (The terminology is a bit confusing here!) for AbuseFilter^[2]. (FYI, AbuseFilter actions are not run in a job queue, since they sometimes need to prevent the action, only change tagging is deferred since it needs the RC saved.) ORES I don't know enough about it to implement but I believe it's run in a job queue so would not be done at the time of save. For bots, it's after save when we get the API request, like rollback. Cenarium (talk) 12:48, 20 October 2016 (UTC)[reply]
    • Thanks for the clarifications. --Halfak (WMF) (talk) 20:26, 20 October 2016 (UTC)[reply]
• Can an RfC even
  snow-close as passed? Because I think this one might qualify. — Gestrid (talk) 08:00, 8 November 2016 (UTC)[reply
  ]
  • @Gestrid: In theory, yes, of course. Happens all the time with content discussions. In practice, something as significant as this should probably run 30 days. ~ Rob₁₃^Talk 08:24, 8 November 2016 (UTC)[reply]
    • @Rob₁₃: I figured. Who knows? Once everyone's done watching the election, this RfC could fail miserably. (I said half-kidding.) — Gestrid (talk) 08:27, 8 November 2016 (UTC)[reply]