Wikipedia:Bots/Requests for approval/MusikBot 10

The following discussion is an archived debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA. The result of the discussion was

Approved.

MusikBot 10

Operator: MusikAnimal (talk · contribs · SUL · edit count · logs · page moves · block log · rights log · ANI search)

Time filed: 05:55, Monday, January 9, 2017 (UTC)

Automatic, Supervised, or Manual: Supervised

Programming language(s): Ruby

Source code available: GitHub

Function overview: Move BLPs created by Sander.v.Ginkel to the draftspace

Links to relevant discussions (where appropriate): User:Aymatth2/SvG clean-up, specifically Special:Permalink/759048541#Hang on a second

Edit period(s): One time run

Estimated number of pages affected: ~16,104 page moves

Exclusion compliant (Yes/No): No

Already has a bot flag (Yes/No): Yes

Function details: The ANI discussion on why this is being done can be found at Wikipedia:Administrators' noticeboard/IncidentArchive941#User:Fram. The result of that discussion, among other things, was to re-evaluate problematic BLPs created by Sander.v.Ginkel. The issue is there are over 16,000 of them, so after some discussion at User:Aymatth2/SvG clean-up involved editors have decided they want to move these pages to the draftspace. There they can be manually reviewed for inclusion, following the clean-up guidelines. The BLPs were identified by quarry:query/15207 and spot-checked by myself and Aymatth2. All redirects will be suppressed (see Special:ListGroupRights, bots possess the suppressredirect permission), and categories will be deactivated (e.g. change [[Category:Living people]] to [[:Category:Living people]]. The whole process will be supervised, but ideally at a high-rate to reduce total runtime. For perspective, one page move per second would still take over 4 hours to complete. A slower rate is perfectly fine, but mind you this is a straightforward task that does not involve any special logic — MusikAnimal ^talk 05:55, 9 January 2017 (UTC)[reply]

Discussion

Comment: One page per second, or even ten per minute to allow up to one day for this work to happen, should not be a big deal. I don't think there is need for a big hurry to get them moved, given how long it will take to review these articles. Get it done, sure, but don't worry about how long it takes. – Jonesey95 (talk) 06:33, 9 January 2017 (UTC)[reply]

The closing admin of the ANI thread, @Avraham: wanted to give only a week for the entire review while the articles were in mainspace due to the high probability of BLP violations. One day is not a big deal, but longer than that is likely a problem. Tazerdadog (talk) 06:39, 9 January 2017 (UTC)[reply]

I'm OK with one day too, but I will at some point have to let the bot roam free while I sleep :) Again this super straightforward. The only thing I can think of that could go wrong is some encoding issue with weird page titles. I have a set of test cases I use in developing scripts/bots for this very purpose, and it is able to handle those properly. It'd make me feel better if I monitor the full run, but I still think the likelihood of error is slim. Doing this in chunks – maybe half one day, the other half the next day – is also perfectly fine — MusikAnimal ^talk 07:28, 9 January 2017 (UTC)[reply]

Apologies if I'm stating the obvious, but the edit summary for each move needs to be crystal clear on why it's being done. While the ANI did take up a large chunk of that page, with several contributors, and projects such as Olympics, Cycling, WIR, etc, have been told about the moves/deletions, there will be lots of editors for one reason or another, who know nothing about why it's being done. With plenty of different sports covered, no doubt several people's watchlists will light up when things start moving out of mainspace! Lugnuts ^{Precious bodily fluids} 11:17, 9 January 2017 (UTC)[reply]

Yeah. I doubt we can explain it well enough in an edit summary honestly. We should make a page that edit summaries can link to that gives the brief version of events to this point, and invites users to help with the cleanup. Tazerdadog (talk) 14:25, 9 January 2017 (UTC)[reply]

Someone could summarize the ANI discussion and outcome in a section on this page, then the bot edits could link here. It may not be the usual way of doing things, but creating a third page would also be a little strange. – Jonesey95 (talk) 16:29, 9 January 2017 (UTC)[reply]

Make the edit summary: "Moved to draft space per User:Aymatth2/SvG clean-up/Guidelines". That points to the other discussions, and also explains how the article can be salvaged. Aymatth2 (talk) 18:14, 9 January 2017 (UTC)[reply]

Sounds good to me. Lugnuts ^{Precious bodily fluids} 08:21, 10 January 2017 (UTC)[reply]

Done [1] — MusikAnimal ^talk 19:09, 10 January 2017 (UTC)[reply]

I am more familiar with COBOL/MVS, but it certainly looks as though that will do the job in a very straightforward way. As soon as the move is finished, I can clean up the lists and notify the projects and users. Aymatth2 (talk) 20:53, 10 January 2017 (UTC)[reply]

What's the progress on approval for this/starting the moves? Thanks. Lugnuts ^{Precious bodily fluids} 12:30, 11 January 2017 (UTC)[reply]

(Non-BAG comment) Having reviewed the ANI and discussions/pages in Aymatth2's userspace, this should be approved for a small trial to verify that redirects won't remain and there aren't any encoding issues. — JJMC89 (T·C) 17:06, 11 January 2017 (UTC)[reply]

{{BAGAssistanceNeeded}} This apparently is a bit pressing given the potential BLP violations floating around, and the nature of the task should make for an easy review. Many thanks — MusikAnimal ^talk 20:19, 11 January 2017 (UTC)[reply]

Approved for trial (25 edits). Please provide a link to the relevant contributions and/or diffs when the trial is complete. — xaosflux ^Talk 02:41, 13 January 2017 (UTC)[reply]

So after finally figuring out I needed to change the applicable grants via Special:BotPasswords in order to get suppressredirect to work, I made a single page move (including the talk page). Then I realized we should probably be deactivating the categories? E.g. change [[Category:Living people]] to [[:Category:Living people]]. I'm near certain we want to do this, but will run it by you all first. Xaosflux, I also realized by moving the talk pages too we'll likely end up with an even number of edits, so I will do 24 instead of 25 — MusikAnimal ^talk 08:20, 13 January 2017 (UTC)[reply]

Please do disable the categories if possible. Tazerdadog (talk) 08:28, 13 January 2017 (UTC)[reply]

The bot ideally should do everything that can easily be done in the step 2 (Tidy Up) found here:User:Aymatth2/SvG_clean-up#Saving_an_article_from_deletion. Tazerdadog (talk) 08:43, 13 January 2017 (UTC)[reply]

Trial complete. Worked out to be 25 edits [2]. A typo left out a colon the first time it deactivated the categories [3] but easily fixed and it worked fine for the rest of the run. I can help with the "tidy up" step User:Aymatth2/SvG_clean-up#Saving_an_article_from_deletion, but we should talk about that more... For starters I think you want {{draft article}} and not {{userspace draft}}. The main issue is stubbing out non-free imagery. I think I can have the bot detect if it's non-free, but backing up the image to the file system is the bigger concern. I'm not going to do that on Tool Labs, and I'm pretty sure non-free imagery is not permissible there anyway. Considering many of the 16,000+ pages have (possibly non-free) images, I'm guessing that's going to take a toll on the disk space of my local machine. You may have to do without them entirely, or go through the drafts manually and deal with the images. Finally, it's unclear if we'd want to remove all maintenance templates? Perhaps that part should be left to manual review, too. Please advise — MusikAnimal ^talk 08:55, 13 January 2017 (UTC)[reply]

Yes, {{draft article}} and not {{userspace draft}}. Good catch. It is not permissible to have fair use files in draft articles because of copyright concerns. If the bot simply stubbed out non-free images, and then listed the articles that contained them somewhere, I could review them quickly enough to prevent the file from being deleted. In the worst case scenario, deleting all of the non-free images is clearly the lesser of the evils. The articles are being reviewed individually, and we do not want to clog up the queue of anyone working through a maintenance template with these articles. I'd therefore be inclined to say yes, kill all maintenance templates, but I very easily could be wrong on this point. Tazerdadog (talk) 09:08, 13 January 2017 (UTC)[reply]

We could also do as the clean up guide says and change [[File:Image]] to [[:File:Image]] (or similar, for infoboxes), allow the image to be procedurally deleted, and later request undeletion as needed. Also bear in mind these new bot requests will take some time to code, and involve a more lengthy trial. You may wish to weigh out the concern of the potential BLP violations lingering in the mainspace versus handing the clean up tasks manually. In my opinion the maintenance templates may be helpful when reviewing, and I think non-free imagery in the draft space is not quite as bad as live BLP violations indexed by search engines — MusikAnimal ^talk 09:15, 13 January 2017 (UTC)[reply]

Yes, The BLP violations outweigh all of those concerns right now. Let's get everything moved, and once they are moved, we can talk about maintenance templates and similar. I just did a spot check - I'd be surprised if 10% of the articles contained an image, and further surprised if 1% of them contained a non-free one. Based on that, I think the existing processes are good enough for images and the bot shouldn't try to deal with them. Let's keep it simple Tazerdadog (talk) 09:30, 13 January 2017 (UTC)[reply]

The test looks fine to me. No redirects left behind, categories disabled. I would not worry about a few drafts linking to non-free images, or some non-free images being lost. These are tiny details. I say, let's go. Does this need another approval? Aymatth2 (talk) 13:27, 13 January 2017 (UTC)[reply]

Looks good - thanks for the work so far. @Aymatth2: - how do these lists work once the article is moved from mainspace and the categories are disabled? For example, the Olympic cyclist list has already dropped to 148 after the 25-page trial move. Lugnuts ^{Precious bodily fluids} 15:01, 13 January 2017 (UTC)[reply]

I did not think of that. I will make files holding copies of the existing lists. Are there others that should be added before we lose the categories? Aymatth2 (talk) 15:40, 13 January 2017 (UTC)[reply]

That's great. No, I don't think there are any others. They're pretty comprehensive as they stand. Thanks. Lugnuts ^{Precious bodily fluids} 19:20, 13 January 2017 (UTC)[reply]

@Lugnuts: Thanks. I have all the lists downloaded, and am reformatting and uploading them as files, a tedious job. In User:Aymatth2/SvG clean-up/Guidelines#Lists I am putting the file name below the query, so they can be compared if anyone wants to check before the mass move is done. There are a few some small discrepancies in counts, perhaps partly from the trial run, partly from other changes to the articles. I am not going to worry about that. Aymatth2 (talk) 20:03, 13 January 2017 (UTC)[reply]

@Lugnuts and Aymatth2: So are we OK to move forward with the bot, without doing any of the cleanup tasks (except deactivating categories)? — MusikAnimal ^talk 20:05, 13 January 2017 (UTC)[reply]

Aymatth2 - appreciate your work with this. If you could do the main Olympic one, that would be great. MusikAnimal - yes, happy for things to get going and to start the 90-day clock! Thanks. Lugnuts ^{Precious bodily fluids} 20:09, 13 January 2017 (UTC)[reply]

I am brain dead from editing these lists.

@MusikAnimal: Yes - no need for any improvements to the bot. Good to go as far as I am concerned. Aymatth2 (talk) 21:26, 13 January 2017 (UTC)[reply]

@Tazerdadog: Any chance that after the move you could fix the files at User:Aymatth2/SvG clean-up/Guidelines#Lists, e.g. User:Aymatth2/SvG clean-up/Runners. Keep the same file names but change all [[ to [[Draft:? This should be done before we notify the editors and projects. Thanks, Aymatth2 (talk) 21:26, 13 January 2017 (UTC)[reply]

@Lugnuts: I have turned all the lists into files. Once the mass move has run and the lists have been fixed, would you be willing to notify all the projects and users and ask for help with salvage? I would say any project or userid in User:Aymatth2/SvG clean-up and User:Aymatth2/SvG clean-up/Guidelines, anyone else you can think of. It will probably be a bit chaotic, but the more help the better. Thanks, Aymatth2 (talk) 21:26, 13 January 2017 (UTC)[reply]

Glad to hear the bot is good to go on your end. If we could keep unrelated discussion outside the BRFA that would be preferable, perhaps User talk:Aymatth2/SvG clean-up? Thanks :) — MusikAnimal ^talk 21:29, 13 January 2017 (UTC)[reply]

(TL;DR) See bot trial results above. We're only moving the pages to the draft space without redirect, and deactivating the categories — MusikAnimal ^talk 23:08, 13 January 2017 (UTC)[reply]

Thanks once again both of you for all your help on this. I think we're good to go with the bot. Lugnuts ^{Precious bodily fluids} 13:07, 14 January 2017 (UTC)[reply]

@MusikAnimal: - Do you know when the bot will start now the trial is complete? Thanks. Lugnuts ^{Precious bodily fluids} 18:30, 15 January 2017 (UTC)[reply]

As soon as we get approval, will request BAG attention — MusikAnimal ^talk 19:00, 15 January 2017 (UTC)[reply]

{{BAGAssistanceNeeded}} Again a tad urgent given the possibility of BLP violations lingering in the mainspace. See above for trial results. The bot will only move the pages to the draft space then deactivate the categories — MusikAnimal ^talk 19:00, 15 January 2017 (UTC)[reply]

@Xaosflux: Sorry to bug you! Any chance you could look this over? Should be easy peasy :) — MusikAnimal ^talk 00:48, 18 January 2017 (UTC)[reply]

Approved for extended trial (250 edits). Please provide a link to the relevant contributions and/or diffs when the trial is complete. I'd like to see a larger run to ensure other odd use cases don't come up. Please post results when done. — xaosflux ^Talk 03:41, 18 January 2017 (UTC)[reply]

@MusikAnimal:, @Xaosflux: - all done? Trial complete and a success? Can we now move all of them? Or do we have to wait another week/10 days or more for someone from this bot approval group to wake up? Lugnuts ^{Precious bodily fluids} 08:03, 19 January 2017 (UTC)[reply]

Sorry the process has been slow, but trials are important, and bot approval is a requirement. No issues with the new trial, so hopefully we'll get this approved soon. Thanks for your patience — MusikAnimal ^talk 10:56, 19 January 2017 (UTC)[reply]

Trial complete. With 252 edits. Pages like Draft talk:Mieke Kröger with special characters (special to the Ruby gem I'm using, anyway) were properly handled. There is an issue, though... I'm guessing the bot is moving some pages back to the draft space that have already been re-reviewed under this clean up project. To get around that, we need to stop doing anymore reviews until the bot task is approved and completed (hopefully next run)... I say this because there were a few pages the bot was unable to move because the target draft page was not a redirect to the source page, such as [4]. Here a new-ish user (who probably shouldn't be doing reviews) reviewed the page and erroneously moved it to the Wikipedia namespace. That explains why the bot couldn't move Aafke Eshuis to Draft:Aafke Eshuis, but it also tells me we're going in circles. So if things look okay to Xaosflux, hopefully we can get a "move on" (pun intended) so that you all can start reviewing again — MusikAnimal ^talk 10:56, 19 January 2017 (UTC)[reply]

Approved. with ramp up schedule. I understand the urgency, but will give slight pauses in the event of other errors.

2000 edits, 12 hour hold
2000 edits, 12 hour hold
5000 edits, 12 hour hold
rest of job — xaosflux ^Talk 12:10, 19 January 2017 (UTC)[reply]

The above discussion is preserved as an archive of the debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA.