View source for Wikipedia:Bots/Requests for approval/WildBot 8

<noinclude>[[Category:Expired Wikipedia bot requests for approval|WildBot 8]]</noinclude><div class="boilerplate metadata" style="background-color:
#DEDACA; margin:2em 0 0 0; padding:0 10px 0 10px; border:1px solid #AAAAAA;">
:''The following discussion is an archived debate. <span style="color:red">'''Please do not modify it.'''</span> To request review of this BRFA, please start a new section at [[WT:BRFA]].'' The result of the discussion was [[File:Symbol neutral vote.svg|20px|alt=|link=]] '''Request Expired'''.<!-- from Template:Bot Top-->
==[[User:WildBot|WildBot 8]]==
{{Newbot|WildBot|8}}
<!-- ^^^ Ignore everything above this line ^^^ -->

<!-- Leave this alone -->
'''Operator:''' {{botop|Josh Parris}}

<!-- And leave this alone -->
'''Time filed:''' 23:47, Friday December 6, 2013 ([[UTC]])

<!-- Automatic: edits are not reviewed, except when issues arise -->
<!-- Supervised: all edits are reviewed soon after being made -->
<!-- Manual: all edits are reviewed before being made -->
'''Automatic, Supervised, or Manual:''' Automatic

<!-- Programming Language: The computer language that this bot will be written in. E.g. Python, Java, C, VB, AutoWikiBrowser. NOT English etc. -->
'''[[Programming language]](s):''' Python, wikitools

<!-- Source code available: e.g. a link to the source code, "Standard pywikipedia"/"AWB"/etc. Be sure the bot account's password is not given out! -->
'''[[Source code]] available:''' Sure, prompt me

<!-- A SHORT bot function overview (max 2–3 lines); place the in-depth explanation in the "function details" section below-->
'''Function overview:''' Replace broken urls to *.thecanadianencyclopedia.com with working ones to thecanadianencyclopedia.ca

<!-- Bot tasks require consensus in order to be approved. Please list any relevant discussions here to indicate consensus for the task. If such input is not necessary (for instance, a task that is duplicating or closely matching an existing bot) leave this blank-->
'''Links to relevant discussions (where appropriate):''' [[Wikipedia:Bot requests/Archive 57#URL updates for The Canadian Encyclopedia]]

<!-- e.g. Continuous, daily, one time run, etc. -->
'''Edit period(s):''' one run

<!-- Should be a reasonable guess as to how many distinct pages you'll be editing. For open-ended tasks, estimate pages per some reasonable time period.-->
'''Estimated number of pages affected:''' ~3468

<!-- Whether the bot skips articles and other pages with the {{bots}} template. If "no", please provide a reason in the Function details or Discussion.-->
'''[[Template:bots|Exclusion compliant]]''' ''(Yes/No)''''':'''  No, one-time run

<!-- For existing bots only-->
'''Already has a bot flag''' ''(Yes/No)''''':''' Yes

<!-- List full and complete function details here. Please be precise and explicit, describing all changes the bot would make. Bots cannot be approved for open-ended tasks, so ensure the details cover all cases. Consider making several BRFAs for tasks with large independent changes. Vague or incomplete details can delay the BRFA process. Straight-forward details and examples will speed up the approval. If you need to modify these details after a discussion starts, it is recommended that you use del tags – <del></del> – to remove text and ins tags – <ins></ins> – for additions (this preserves coherence of the discussion), or show/hide boxes as you see fit. -->
'''Function details:''' swap broken urls for tested good ones.  I have assembled a mapping of certain URL updates for The Canadian Encyclopedia based on lookups into the [[Wayback Machine]] of all external URLs that match *.thecanadianencyclopedia.com and used that to generate and test combinations of URLs against thecanadianencyclopedia.ca until I got a 200-sucess. <del>Links to the home page of the site will be stripped.</del> URLs where I couldn't get a successful hit will be left unchanged. <ins>Variations on the {{tl|dead link}} templates are <del>added or removed to the article to reflect the status of external links; they're only</del> removed for ''thecanadianencyclopedia''. <del>The work parameter of the various cite templates is altered to change hyperlinks into domains.</del></ins>

===Discussion===
<!-- This is not a vote.  It is a discussion -->
{{BotTrial|edits=10}} —&nbsp;<small>&nbsp;[[user:Hellknowz|<span style="color:#B00;">HELL</span>KNOWZ]]&nbsp;&nbsp;▎[[User talk:Hellknowz|TALK]]</small> 22:30, 11 December 2013 (UTC)
:{{BotTrialComplete}} 
:[https://en.wikipedia.org/enwiki/w/index.php?title=Special:Contributions/WildBot&offset=20131212110000&target=WildBot&limit=11 10 Trial edits] in I made a stupid error with my HTML comment not being an actual comment. I fixed all the edits.
:More interesting errors:
:*[https://en.wikipedia.org/enwiki/w/index.php?title=Vernon_Barford&diff=prev&oldid=585669074] "corrected" to a 404. The substitution did what it was told.  The translation list was mis-populated because of a parsing error on http://web.archive.org/web/20110929060526/http://www.thecanadianencyclopedia.com/index.cfm?PgNm=TCE&Params=U1ARTU0000203 where the "writing" section was determined to be the article title.  An inspection of the translation list shows this has not occurred on any other occasion, nor has "author" nor "bibliography" been the target of any translations.  It was just good luck catching this one.  Fixed on a couple of levels - tighter regex matching, plus those headings have been added to the blacklist.
:*[https://en.wikipedia.org/enwiki/w/index.php?title=University_of_Toronto_Quarterly&diff=prev&oldid=585669061] shows a replacement within a ref tag where the link is followed by a {{tlx|dead link|2=date=December 2013}}. Ought I be removing these {{tl|dead link}}s? [[User_talk:Josh Parris|Josh Parris]] 02:44, 12 December 2013 (UTC)
::*Do I get this right -- you are comparing the actual page content on wayback archived version to find matches? Or title?
::*Yes, you should remove {{tl|dead link}}s after the citation or reference tag if you fix them. —&nbsp;<small>&nbsp;[[user:Hellknowz|<span style="color:#B00;">HELL</span>KNOWZ]]&nbsp;&nbsp;▎[[User talk:Hellknowz|TALK]]</small> 13:16, 13 December 2013 (UTC)
::*:The technique I'm using for translating from the old URL to the new one is:
::*:#Check for a 302 redirect sometime in 2012.  The redirect will be to a URL similar to what's used now, with quite a few variations - a trailing slash may or may not be required, the order of words may have changed, parts of the path might have been moved around.
::*:#Failing that, the Wayback Machine's copy will have an article title, which might be transformable in various ways into the corresponding URL in the new website
::*:All I'm doing is checking for a 200 status code to confirm a match - do you think I ought to be doing something less naive?
::*:I'll get onto removing deadlink tags; it might be easy, or perhaps not. [[User_talk:Josh Parris|Josh Parris]] 11:48, 14 December 2013 (UTC)
:::I think that's good enough -- I don't think there would be obvious false positives, especially if you use their own 302s. —&nbsp;<small>&nbsp;[[user:Hellknowz|<span style="color:#B00;">HELL</span>KNOWZ]]&nbsp;&nbsp;▎[[User talk:Hellknowz|TALK]]</small> 12:03, 14 December 2013 (UTC)

{{BotExtendedTrial|edits=10}} —&nbsp;<small>&nbsp;[[user:Hellknowz|<span style="color:#B00;">HELL</span>KNOWZ]]&nbsp;&nbsp;▎[[User talk:Hellknowz|TALK]]</small> 12:03, 14 December 2013 (UTC)
:{{BotTrialComplete}} Wow, that expanded the source code dramatically. I selected ten articles that had {{tl|dead link}} and canadianencyclopedia urls. Performed [https://en.wikipedia.org/enwiki/w/index.php?title=Special:Contributions/WildBot&offset=20131217060000&target=WildBot&limit=10 10 trial edits], highlights include:
:*[https://en.wikipedia.org/enwiki/w/index.php?title=Karate&diff=prev&oldid=586451198] Shows the {{tl|dead link}} template being removed for the repaired link
:*[https://en.wikipedia.org/enwiki/w/index.php?title=Moncton&diff=prev&oldid=586452097] Shows how subst doesn't work for bots (fixed)
:*[https://en.wikipedia.org/enwiki/w/index.php?title=Conscription&diff=prev&oldid=586456165] demonstrates removing a URL from a cite template's work parameter
:So, it seems all went well. [[User_talk:Josh Parris|Josh Parris]] 06:55, 17 December 2013 (UTC)

::Okay, these look good, but that's quite a range of functionality. —&nbsp;<small>&nbsp;[[user:Hellknowz|<span style="color:#B00;">HELL</span>KNOWZ]]&nbsp;&nbsp;▎[[User talk:Hellknowz|TALK]]</small> 13:47, 18 December 2013 (UTC)

{{BotExtendedTrial|edits=20}} More trial since the addition of code and just a larger sample. —&nbsp;<small>&nbsp;[[user:Hellknowz|<span style="color:#B00;">HELL</span>KNOWZ]]&nbsp;&nbsp;▎[[User talk:Hellknowz|TALK]]</small> 13:47, 18 December 2013 (UTC)
:{{BotTrialComplete}} with [https://en.wikipedia.org/enwiki/w/index.php?title=Special:Contributions/WildBot&offset=20131219010000&target=WildBot&limit=20 results here].  Points of note:
:*[https://en.wikipedia.org/enwiki/w/index.php?title=Glenn_Gould&diff=prev&oldid=586711316] has the bot swapping out a dead url for text, which would be fine except this is a url= field. I've removed this functionality from the bot and will leave it to humans to clean up these urls. But [https://en.wikipedia.org/enwiki/w/index.php?title=Leonard_Cohen&diff=prev&oldid=586718585#cite_ref-macleansbroke_37-0] shows I removed it wrong; I should have detected those URLs and done nothing, rather than treating them as any other URL. Fixed.
:*[https://en.wikipedia.org/enwiki/w/index.php?title=Andr%C3%A9_Laplante&diff=prev&oldid=586711618] has the bot making supplemental fixes but not the main fix of swapping dead urls. This was due to a logic bug in the code to detect null edits - fixed.
:I stand ready for another trial. [[User_talk:Josh Parris|Josh Parris]] 00:59, 19 December 2013 (UTC)

{{BotExtendedTrial|edits=50}} without removing external links from {{para|work}}. —&nbsp;<small>&nbsp;[[user:Hellknowz|<span style="color:#B00;">HELL</span>KNOWZ]]&nbsp;&nbsp;▎[[User talk:Hellknowz|TALK]]</small> 10:28, 19 December 2013 (UTC)
:Functionality altered to reflect this. [[User_talk:Josh Parris|Josh Parris]] 10:40, 19 December 2013 (UTC)
:{{BotTrialComplete}} after [https://en.wikipedia.org/enwiki/w/index.php?title=Special:Contributions/WildBot&offset=20131219113000&target=WildBot&limit=50 50 edits]. Every edit seems fine.
:I did get a scare from [https://en.wikipedia.org/enwiki/w/index.php?title=Jeanne_d%27Arc_au_b%C3%BBcher&diff=prev&oldid=586773726], but looking at http://web.archive.org/web/20120315000000*/http://www.thecanadianencyclopedia.com/index.cfm?PgNm=TCE&Params=U1ARTU0002865 I'm reassured that the bot isn't at fault. [[User_talk:Josh Parris|Josh Parris]] 11:27, 19 December 2013 (UTC)

* [https://en.wikipedia.org/enwiki/w/index.php?title=Pierre_Maillard&diff=prev&oldid=586771858] What's up with first dead link here? Or [https://en.wikipedia.org/enwiki/w/index.php?title=Toronto_Light_Opera_Association&diff=prev&oldid=586772102 here]], although here it ''is'' dead. This happens in many pages, are you now checking unrelated links for 404s (I must have misread this from the <small>ever-changing</small> function details)? —&nbsp;<small>&nbsp;[[user:Hellknowz|<span style="color:#B00;">HELL</span>KNOWZ]]&nbsp;&nbsp;▎[[User talk:Hellknowz|TALK]]</small> 17:17, 21 December 2013 (UTC)

*:Sorry for the delay in responding; my Internet is temperamental right now.
*:The problem with that first link is that it was to an HTML anchor, and anchors weren't being stripped (now fixed). No other links were incorrectly marked as dead.
*:As an aside, do you have any insight as to why cURL and my browser agree that a 404 is returned for http://www.lethbridge.ca/NR/rdonlyres/D4CEB98B-9F18-4786-870D-84A06E1533FC/310/LethbridgeProfile2003.pdf yet Python's httplib thinks it's a 200?
*:Yes, I figured checking all URLs was a value-add during swapping the Canadian URLs, given I had to check the deadurls weren't dead - you know, "free functionality". [[User_talk:Josh Parris|Josh Parris]] 04:42, 26 December 2013 (UTC)
**It says that because "checking 404 is not an easy task". None of our 404-checking and archiving bots (iuncluding mine) are running simply due to all the continuous issues that they have and all the ingenious ways web developers break them. That's why I didn't think you were also checking ''all'' dead links. That will extend this BRFA and trials a lot and I really recommend this be a separate task (<small>or I'll go mad</small>). Not to mention, you cannot check a link once, you need to come back in a week or so and check it again or there will be tons of false positives on temporarily 404ed sites. In your case, there was possibly a redirect (one of several ways to do it) or a different page version served as there might be an agent or referrer check or cookie requirement or many of the subtle HTTP protocol options. It could just be some broken headers or inconsistencies between curl and your typical browsers. Not setting up a cookie container has led to many sites failing on me. —&nbsp;<small>&nbsp;[[user:Hellknowz|<span style="color:#B00;">HELL</span>KNOWZ]]&nbsp;&nbsp;▎[[User talk:Hellknowz|TALK]]</small> 20:02, 26 December 2013 (UTC)
**:Gotcha.  Pulled 404 checking for anything other than the target URLs. [[User_talk:Josh Parris|Josh Parris]] 03:45, 27 December 2013 (UTC)

{{BotExtendedTrial|edits=20}} —&nbsp;<small>&nbsp;[[user:Hellknowz|<span style="color:#B00;">HELL</span>KNOWZ]]&nbsp;&nbsp;▎[[User talk:Hellknowz|TALK]]</small> 10:48, 27 December 2013 (UTC)
:Sorry for the delay; flaky Internet. {{BotTrialComplete}} After [https://en.wikipedia.org/enwiki/w/index.php?title=Special:Contributions/WildBot&offset=20131228090000&target=WildBot&limit=20 20 edits], every edit seems fine. [[User_talk:Josh Parris|Josh Parris]] 09:17, 28 December 2013 (UTC)
::I've got solid Internet under my feet now, so {{tl|BAGAssistanceNeeded}} [[User_talk:Josh Parris|Josh Parris]] 20:39, 5 January 2014 (UTC)

*[https://en.wikipedia.org/enwiki/w/index.php?title=Ben_Johnson_%28sprinter%29&diff=prev&oldid=588045824] -- [http://www.thecanadianencyclopedia.ca/en/article/macleans/drug-scandal-hits-us-athletes] doesn't look like the right one
*[https://en.wikipedia.org/enwiki/w/index.php?title=Eva_Rose_York_Bible_Training_and_Technical_School_for_Women&diff=prev&oldid=588045829] -- same [http://www.thecanadianencyclopedia.ca/en/article/eva-rose-york-emc/]
*[https://en.wikipedia.org/enwiki/w/index.php?title=February_1999_Kurdish_protests&diff=prev&oldid=588045905] -- same [http://www.thecanadianencyclopedia.ca/en/article/macleans/kurdish-terrorist-captured] (macleans again)
*[https://en.wikipedia.org/enwiki/w/index.php?title=The_Queensway_%E2%80%93_Humber_Bay&diff=prev&oldid=588045886] -- broken link
Note that I haven't (yet) gone through previous trials link by link. —&nbsp;<small>&nbsp;[[user:Hellknowz|<span style="color:#B00;">HELL</span>KNOWZ]]&nbsp;&nbsp;▎[[User talk:Hellknowz|TALK]]</small> 21:59, 5 January 2014 (UTC)

:If only there was an exasperated sigh template I could invoke here.
:The Ben Johnson (and Kurdish protest) edits show the "check for a 200 status" rule isn't adequate.  I'll work up something more robust in the face of this.
:The Eva Rose York edit is actually fine.
:The [[The Queensway – Humber Bay]] edit is particularly galling, as running the list generator against the page today pulls up the 404 and can't resolve it, but going to the URL in the article redirects to a valid article. The site operator has not only 404'd their old URL, they've made the older one work by redirecting it to their new one.  I'm going to have to throw away my old translation list and regenerate it.
:I'll ping back once I've made the necessary code changes.  Expect a two week delay. [[User_talk:Josh Parris|Josh Parris]] 21:40, 6 January 2014 (UTC)
::That fix was easier than I thought. 
::It seems something similar to the mcleans thing happened with French articles, so I already had code to simply strip it out.
::I've coded up a fix to the 404.
::I'm going to review all the edits the bot made since the start of time and confirm they correlate to what the bot would now do, and repair anything that's wrong. [[User_talk:Josh Parris|Josh Parris]] 14:49, 8 January 2014 (UTC)

{{OperatorAssistanceNeeded|D}} Anything new about this taske? [[Special:Contributions/46.107.88.236|46.107.88.236]] ([[User talk:46.107.88.236|talk]]) 16:45, 24 January 2014 (UTC)
: {{Ping|Josh Parris}} Any progress? <span style="background: turquoise; font-family: 'Segoe Script'">([[User talk:Josve05a|'''t''']])&nbsp;[[User:Josve05a|<span style="color:white;">'''Josve05a'''</span>]]&nbsp;([[Special:Contributions/Josve05a|'''c''']])</span> 13:12, 2 April 2014 (UTC)

{{BotExpired}} --[[User:Slakr|<span style="color:teal;font-weight:bold;">slakr</span>]]<small><sup>\&nbsp;[[User talk:Slakr|talk]]&nbsp;/</sup></small> 07:03, 12 April 2014 (UTC)
:''The above discussion is preserved as an archive of the debate.  <span style="color:red">'''Please do not modify it.'''</span> To request review of this BRFA, please start a new section at [[WT:BRFA]].''<!-- from Template:Bot Bottom --></div>