Wikipedia:Link rot/URL change requests
This page is for requesting modifications to URLs, such as marking dead or changing to a new domain. Some bots are designed to fix link rot; they can be notified here. These bots include InternetArchiveBot and WaybackMedic. This page can be monitored by bot operators from other language wikis since URL changes are universally applicable.
finlex.fi
 | This section is pinned and will not be automatically archived. |
Finlex.fi URLs aren't dead but for some reason InternetArchiveBot keeps adding archived URLs for them. This was brought up at meta:User_talk:InternetArchiveBot#Finlex.fi_URLs_aren't_dead a month ago: Bot's edits: [1], [2], [3]. Some URLs it tagged as dead but are actually working: [4], [5], [6].
Those finlex.fi URLs that now have both a working URL and an archive URL should be tagged with the |url-status=live tag, and could someone try to tell IABot that Finlex is live? Thanks. 2001:14BA:9C94:9A00:E866:DADA:1085:E3D9 (talk) 09:28, 17 March 2024 (UTC)
- Just noticed that this same issue is being discussed at fi.wikipedia: fi:Wikipedia:Kahvihuone_(tekniikka)#Botti_hakee_arkistosta_kumottuja_lakeja 2001:14BA:9C94:9A00:E866:DADA:1085:E3D9 (talk) 09:41, 17 March 2024 (UTC)
- The site has a "Are you human?" check box (CloudFlare). This is causing the bot to think it's a dead site. I logged into iabot.org and changed the domain to "Subscription" status and that will cause the bot to avoid this domain, it won't set live or dead. My bot WaybackMedic has capabilities to bypass CloudFlare. I can try to process this domain and see what happens. My bot also has a feature "make live" ie. convert a citation from dead to live state. Unfortunately my bot only works on English Wikipedia. I'll let you know what happens. -- GreenC 15:13, 17 March 2024 (UTC)
- Unfortunately, this site has maximum security enabled, none of my tools can get through. It started happening in late January 2024. I don't know what to do because no bot is able to determine if a link is live or dead. And no archive service such as WaybackMachine is able to archive a page. Only humans can get through, and they need to solve a captcha. It might be worthwhile waiting to see if they relax security in the future, since this is a recent development. -- GreenC 00:40, 19 March 2024 (UTC)
- @GreenC: Before this section gets archived and if it's easy/fast to check, can you check if this is still the case, i.e. that the site still has the maximum security enabled and no tool/bot can get through? Thank you. 85.76.109.152 (talk) 06:21, 2 June 2024 (UTC)
When going to [7] it still asks "Are you human?" with the CloudFlare security tag at the bottom. This is a feature of CloudFlare service, clients have the option to enable, it's the highest level of security. I'm not aware of a tool that can bypass. What I will do is set a reminder in 6 months to check again and post the results here. I use W-Ping which posts a reminder in the watchlist at whatever time in the future with a custom message. -- GreenC 16:06, 2 June 2024 (UTC)
- @GreenC: Before this section gets archived and if it's easy/fast to check, can you check if this is still the case, i.e. that the site still has the maximum security enabled and no tool/bot can get through? Thank you. 85.76.109.152 (talk) 06:21, 2 June 2024 (UTC)
- Unfortunately, this site has maximum security enabled, none of my tools can get through. It started happening in late January 2024. I don't know what to do because no bot is able to determine if a link is live or dead. And no archive service such as WaybackMachine is able to archive a page. Only humans can get through, and they need to solve a captcha. It might be worthwhile waiting to see if they relax security in the future, since this is a recent development. -- GreenC 00:40, 19 March 2024 (UTC)
- The site has a "Are you human?" check box (CloudFlare). This is causing the bot to think it's a dead site. I logged into iabot.org and changed the domain to "Subscription" status and that will cause the bot to avoid this domain, it won't set live or dead. My bot WaybackMedic has capabilities to bypass CloudFlare. I can try to process this domain and see what happens. My bot also has a feature "make live" ie. convert a citation from dead to live state. Unfortunately my bot only works on English Wikipedia. I'll let you know what happens. -- GreenC 15:13, 17 March 2024 (UTC)
smmsport.com
Smmsport.com appears to have been usurped by an online gambling operation masquerading as the original site. Some links, such as [8] and [9], appear to still work and are intact with their original content, while others return 404 errors. But anything linked from the home page is fake. --Paul_012 (talk) 11:09, 29 July 2024 (UTC)
- User:Paul 012: 400+ pages. I'm not seeing gambling pages. Can you find examples? -- GreenC 15:27, 29 July 2024 (UTC)
- They're somewhat insidiously inserted into the first top navigation menu. [10] for example is a link farm advertising gambling sites. --Paul_012 (talk) 15:34, 29 July 2024 (UTC)
- Ahh I see. This is somewhat unusual case of WP:USURPSOURCE. Probably we need an edit filter to prevent editors from adding more links they believe are legitimate, but actually insidious spam (ie. MediaWiki_talk:Spam-blacklist#Proposed_additions). And the existing links usurped by WaybackMedic (ie. this URLREQ). As the primary discoverer, can you make the Spam Blacklist request? -- GreenC 16:54, 29 July 2024 (UTC)
- I added it to the usurpation queue for WaybackMedic Special:Diff/1236486118/1237406269 -- GreenC 16:58, 29 July 2024 (UTC)
- Thanks. I'm not sure about blacklisting, as their old articles could still be useful references. Also, upon closer look, it seems the situation looks more like a hijacking rather than usurpation? Checking the Wayback Machine, the last good version of the home page was archived on 2023-08-13, before the site went down and showed a domain for sale notice. It came back on 2024-06-15, appearing mostly the same as it last did, but by the next archival on 2024-07-02 the gambling links had been inserted into the navigation menu, and the articles linked from the home page had been altered to show a date of 23 May 2024. --Paul_012 (talk) 14:27, 30 July 2024 (UTC)
- The spam blacklist prevents adding new links. Since they appear to have legitimate content, this is a problem editors unknowingly adding new links into Wikipedia, that they found with Google or whatever. It is a classic case of WP:USURPSOURCE. It really needs to be blocked. The old links will be kept and converted to usurped ie. changed to archive URLs, and the source URL no longer hot linked. -- GreenC 14:45, 30 July 2024 (UTC)
- Block request: MediaWiki_talk:Spam-blacklist#smmsport.com -- GreenC 15:54, 31 July 2024 (UTC)
- The spam blacklist prevents adding new links. Since they appear to have legitimate content, this is a problem editors unknowingly adding new links into Wikipedia, that they found with Google or whatever. It is a classic case of WP:USURPSOURCE. It really needs to be blocked. The old links will be kept and converted to usurped ie. changed to archive URLs, and the source URL no longer hot linked. -- GreenC 14:45, 30 July 2024 (UTC)
- Thanks. I'm not sure about blacklisting, as their old articles could still be useful references. Also, upon closer look, it seems the situation looks more like a hijacking rather than usurpation? Checking the Wayback Machine, the last good version of the home page was archived on 2023-08-13, before the site went down and showed a domain for sale notice. It came back on 2024-06-15, appearing mostly the same as it last did, but by the next archival on 2024-07-02 the gambling links had been inserted into the navigation menu, and the articles linked from the home page had been altered to show a date of 23 May 2024. --Paul_012 (talk) 14:27, 30 July 2024 (UTC)
- They're somewhat insidiously inserted into the first top navigation menu. [10] for example is a link farm advertising gambling sites. --Paul_012 (talk) 15:34, 29 July 2024 (UTC)
Done - Bot Results: Batch #13 -- GreenC 14:30, 26 August 2024 (UTC)
fortblissbugle.com
fortblissbugle.com has been usurped by a gambling website. One example is http://fortblissbugle.com/german-air-force-train-at-fort-bliss/ from Fort Bliss.
While, this claims that it's moved to an army website, that website's news archive only goes back to October 24, 2019, a week before the fortblissbugle went offline. Just searching a handful of titles, I can't find anywhere where individual stories are hosted.
46 pages GrapesRock (talk) 18:53, 30 July 2024 (UTC)
- Page says JUDIKING88 at the top. Judi is Indonesian for gambling. Part of the global judi empire. Added to WP:JUDI for later usurpation Special:Diff/1237845244/1238103384 -- GreenC 04:19, 2 August 2024 (UTC)
Done - Bot Results: Batch #13 -- GreenC 14:30, 26 August 2024 (UTC)
Can this be run in Tewiki?
@User:GreenC, In Tewiki, we have more than 10,400 pages in the category CS1 errors: archive-url. Almost 99% of these are "timestamp mismatch" errors. Can you plesase run WaybackMedic_2.5 to correct the error in these pages? Thank you. __ Chaduvari (talk) 15:59, 31 July 2024 (UTC)
- Ahh. I'd like to, but I am not setup for other wikis very difficult. The CS1 error: archive-url is across most wikis. Let me think about it because it's a growing problem. It might be I can process, but only some English-language templates like
{{cite web}}that use English-language parameters like|archive-url=. GreenC 19:25, 31 July 2024 (UTC)- Hi GreenC, in tewiki, this template, like many others, use English parameters and templates only. This policy was kept to ensure future compatibility. Thanks. __ Chaduvari (talk) 09:29, 12 August 2024 (UTC)
- User:Chaduvari, I could try some tests for Telugu Wiki. Can you help me get bot flag permissions for User:GreenC bot? I don't know where to start to ask permission. -- GreenC 18:19, 12 August 2024 (UTC)
- @GreenC, you can raise the request at te:వికీపీడియా:Bot/Requests for approvals.__ Chaduvari (talk) 23:40, 12 August 2024 (UTC)
- I made a request for approval. -- GreenC 02:30, 13 August 2024 (UTC)
- User:Chaduvari, I have not forgotten about this. Have many other projects. Can you tell me what kinds of date formats might exist (date month year, periods or slashes etc) and what Teluga language months? Some examples. -- GreenC 16:56, 26 September 2024 (UTC)
- @GreenC, you have been quick in responding to our request. In fact, we delayed in giving the bot flag.
- The date formats confirm to those in enwiki. 2024-09-27 and 27 September 2024 are the most widely used ones. The month names are:
- @GreenC, you can raise the request at te:వికీపీడియా:Bot/Requests for approvals.__ Chaduvari (talk) 23:40, 12 August 2024 (UTC)
- User:Chaduvari, I could try some tests for Telugu Wiki. Can you help me get bot flag permissions for User:GreenC bot? I don't know where to start to ask permission. -- GreenC 18:19, 12 August 2024 (UTC)
- Hi GreenC, in tewiki, this template, like many others, use English parameters and templates only. This policy was kept to ensure future compatibility. Thanks. __ Chaduvari (talk) 09:29, 12 August 2024 (UTC)
January జనవరి February ఫిబ్రవరి March మార్చి April ఏప్రిల్ May మే June జూన్ July జూలై August ఆగస్టు September సెప్టెంబరు October అక్టోబరు November నవంబరు December డిసెంబరు
- Please look for ref: "Ayodhyaverdict" at page:te:అయోధ్య వివాదంపై 2019 సుప్రీంకోర్టు తీర్పు. The archive date was incorrect in this citation. In the error message, the given Suggestion has the month name in Telugu. (Please look for the text -"మత సామరస్యాన్ని కాపాడాలని ప్రధాన మంత్రి బహిరంగ అభ్యర్థన చేసారు." I am referring to the first citation [10] after this sentence).
- Thank you __ Chaduvari (talk) 00:26, 27 September 2024 (UTC)
- OK. I can't see the red error message in the Wikitext, but it should be possible to scrape it from the HTML. Will investigate. Thank you. -- GreenC 01:14, 27 September 2024 (UTC)
- The easiest way for me is to convert to ISO eg.
|archive-date=2024-09-24. Most of the problems will probably be archive.today and webcitation.org (if any) so I would check every citation template with one of these archives and then reset the archive-date to ISO format, based on the value in the URL. -- GreenC 16:56, 26 September 2024 (UTC)
User:Chaduvari, the tracking category was reduced from 10,400 to 664 for a 94% reduction. The bot I wrote only fixes mismatches in dates. There are other types of errors tracked in that category that bot does not fix. For example citations with an |archive-date= but no |archive-url= (or other way around). Or citations with |archive-url= but no |url=. These are more complex to automatically fix. -- GreenC 04:03, 2 October 2024 (UTC)
- Wow! Fantastic! @GreenC, thanks for eliminating so many errors.
- Now that the errors are brought down by 94% (My estimate fell short by 5% :-)), we will take care of the |archive-url= and other errors manually.
- Thank you very much. __ Chaduvari (talk) 04:53, 2 October 2024 (UTC)
- In fact the number is brought down to 596! __ Chaduvari (talk) 04:54, 2 October 2024 (UTC)
- User:Chaduvari: You are welcome. It can run automatically, every month or so, to keep the category in check. If you see problems it missed, that it should have caught, let me know. -- GreenC 05:17, 2 October 2024 (UTC)
- Sure, GreenC ! Chaduvari (talk) 05:25, 2 October 2024 (UTC)
- OK it will run each month, on the 2nd day. -- GreenC 02:35, 3 October 2024 (UTC)
- Sure, GreenC ! Chaduvari (talk) 05:25, 2 October 2024 (UTC)
- User:Chaduvari: You are welcome. It can run automatically, every month or so, to keep the category in check. If you see problems it missed, that it should have caught, let me know. -- GreenC 05:17, 2 October 2024 (UTC)
- In fact the number is brought down to 596! __ Chaduvari (talk) 04:54, 2 October 2024 (UTC)
articles.latimes.com
Hello. This is a big request. URLs with articles.latimes.com either redirect to the new URL or don't work:
- Working redirect: this to that at Carleton Varney
- Not working redirect: this should go to here for The Man from Earth. These redirects that don't redirect could use archive copies for the time being as the date format doesn't match the above URL at Carleton Varney.
- Broken redirects: this gives a redirect to the Los Angeles Times's main page for Cedars-Sinai Medical Center. It's a search URL and not an URL to a specific article.
60000 with HTTP/HTTPS. Any non articlespace links can be filtered out. Thank you very much! MrLinkinPark333 (talk) 23:58, 12 August 2024 (UTC)
- MrLinkinPark333: It looks like *.latimes.com is 96,000 pages and articles.latimes.com is 37,000. I could focus on articles.latimes.com (which is a significant project) but I wonder about the other 2/3rds. Are they redirecting also? Maybe I should do articles.* right now to keep the size manageable. -- GreenC 02:00, 13 August 2024 (UTC)
- From the sample checks at Summer Olympic Games, Tampa Bay Buccaneers and 2020 Summer Olympics for latimes.com, these look fine and don't need new URLs. If that changes in the future, I could file a separate request later. MrLinkinPark333 (talk) 22:40, 13 August 2024 (UTC)
- OK great. This job is going fast because the LAT has an exceptionally clean site, rapid response, few dead links. It's mostly just finding the redirect and replacing. I'm happy with how well ghost redirect discovery is working, now noted in the statistics (along with soft-404 stats) starting with this run. -- GreenC 16:05, 14 August 2024 (UTC)
- From the sample checks at Summer Olympic Games, Tampa Bay Buccaneers and 2020 Summer Olympics for latimes.com, these look fine and don't need new URLs. If that changes in the future, I could file a separate request later. MrLinkinPark333 (talk) 22:40, 13 August 2024 (UTC)
Enwiki in multiple batches:
- Batch 1: Checked 3,000 pages and edited 2,935 pages. Moved 4,152 links to a new URL. Resolved 68 ghost redirects. Resolved 25 soft-404s. Removed 2
{{dead link}}templates. Added 8{{dead link}}. Switched 149|url-status=deadto live. Switched 14|url-status=liveto dead. Added 166 archive URLs (142 Wayback). Changed 13 citation metadata fields. - Batch 2: Checked 7,000 pages and edited 6,859 pages. Moved 9,663 links to a new URL. Resolved 143 ghost redirects. Resolved 53 soft-404s. Removed 1
{{dead link}}. Added 21{{dead link}}. Switched 314|url-status=deadto live. Switched 34|url-status=liveto dead. Added 372 archive URLs (276 Wayback). Changed 36 citation metadata. - Batch 3: Checked 26,845 pages and edited 26,251 pages. Moved 36,856 links to a new URL. Resolved 481 ghost redirects. Resolved 195 soft-404s. Removed 5
{{dead link}}. Added 89{{dead link}}. Switched 1,289|url-status=deadto live. Switched 128|url-status=liveto dead. Added 1,329 archive URLs (1,106 Wayback). Changed 140 citation metadata.
IABot: does not support URL moves, redirects are working the bot will consider links live.
Done -- GreenC 20:43, 14 August 2024 (UTC)
Pass 2
- Checked 36,845 pages and edited 658 pages. Moved 379 links to a new URL. Resolved 216 ghost redirects. Resolved 438 soft-404s. Removed 2
{{dead link}}. Added 106{{dead link}}. Switched 199|url-status=deadto live. Added 217 archive URLs (138 Wayback).
-- GreenC 02:29, 4 September 2024 (UTC)
emporis.com
Last processed Sept 2022. Many {{dead links}} added. Since then, archive.today added archives, previously unavailable: Special:Diff/1220029968/1240218179. Re-process cites with dead links (emporis3.auth) -- GreenC 05:38, 14 August 2024 (UTC)
- The domain is technically usurped (ie. Emporis). Has 6,000 pages. Will fix in three steps: 1. add archive URLs on enwiki, as a normal dead domain. 2. Same with IABot DB. 3. Later, usurpify everything in a WP:JUDI batch. -- GreenC 03:50, 16 August 2024 (UTC)
- Step 1: Enwiki: Checked 5,979 pages and edited 1,550 pages. Added 430
{{dead link}}. Switched 265|url-status=liveto dead. Added 1,412 archive URLs (136 Wayback). Changed 1,569 citation metadata. - Step 2: IABot DB: Checked 24,000 links. Updated 23,520 links (set permadead and added new archive URLs). Changes will propagate to 300+ wikis via IABot.
- Step 3: Enwiki: usurpify via JUDI batch. Done - Bot Results: Batch #13 -- GreenC 14:29, 26 August 2024 (UTC)
caspianenvironment.org
Shows a page which relates to Car finance in Australia! I believe I have found and changed all instances in the Articlespace, but placed here in case not! Big Blue Cray(fish) Twins (talk) 16:05, 15 August 2024 (UTC)
- Thanks Big Blue Cray(fish) Twins: that's a usurped site. I added it to the list Special:Diff/1239726206/1240485275 .. it will get special handling during a future batch job. -- GreenC 16:18, 15 August 2024 (UTC)
Done - Bot Results: Batch #13 -- GreenC 14:29, 26 August 2024 (UTC)
bcsportshalloffame.com
Hello. I was looking through the Judi list and saw that bcsportshalloffame.com is there. These links could be converted over to their new url at bcsportshall.com/honoured_member/ Here are examples:
- Generally, any individual inductee's url is the person's first and last name. Here is now there for Andrea Neil.
- Last names at birth are included in the urls. Here is now there for Claire Lovett.
- (M) is ignored for team inductees. For instance, here is now there for Mark Evans (rower).
Just over 100 links. If any don't convert over, let me know and I'll fix them. Thanks! MrLinkinPark333 (talk) 19:50, 16 August 2024 (UTC)
- Hmm.. I've never done something like this. It will require de-usurping, like parsing and removing
{{usurped}}. It's an inevitable situation as old usurped domains are migrated to a new working domain. It's probably more complicated than it seems. Will take a look. -- GreenC 16:53, 17 August 2024 (UTC)
I was able to fix cites on 85 pages. The pages it was unable to edit:
Jack Whent Lorne Loomer Charles Edward Pratt 1934 Women's World Games Burnaby Lake Rowing Club Karen Magnussen Lillian Palmer (athlete) Greg Ion Joe Watson (ice hockey) Donald Arnold Archibald MacKinnon Richard McClure Hugh Fisher (canoeist) Sven Habermann Shea Weber Terry Fox 1934 Women's World Games Mary Frizzell
-- GreenC 02:42, 18 August 2024 (UTC)
- Not bad! It makes sense some of it didn't work as the reference titles for Karen Magnussen and Jack Whent were adjusted a bit. I'll tweak the remaining 13 later. Thanks! MrLinkinPark333 (talk) 02:56, 18 August 2024 (UTC)
- OK. Added five more. This search found them. -- GreenC 04:40, 18 August 2024 (UTC)
- I've updated the 18 that the bot couldn't fix. MrLinkinPark333 (talk) 22:56, 13 September 2024 (UTC)
- User:MrLinkinPark333, great! (just saw). -- GreenC 22:22, 18 November 2024 (UTC)
- I've updated the 18 that the bot couldn't fix. MrLinkinPark333 (talk) 22:56, 13 September 2024 (UTC)
- OK. Added five more. This search found them. -- GreenC 04:40, 18 August 2024 (UTC)
Done -- GreenC 14:31, 26 August 2024 (UTC)
erenow.com
Usurped by gambling (e.g. https://erenow.com/postclassical/the-fears-of-henry-iv-the-life-of-englands-king from Wars of the Roses). For pretty clear cut cases like this, can I just add it to the WP:JUDI list directly?
Only 20 pages. GrapesRock (talk) 15:35, 17 August 2024 (UTC)
- Yes, please! -- GreenC 16:41, 17 August 2024 (UTC)
Done - Bot Results: Batch #13 -- GreenC 14:26, 26 August 2024 (UTC)
www-03.ibm.com
It looks like multiple URLs in this domain soft-404. I'm not sure if there are any that don't. Can some or all of these URLs be marked as dead? I marked www-03
- It might be the best solution is the entire www-03 is dead. Will take a look. -- GreenC 22:59, 22 August 2024 (UTC)
- Thank you! McYeee (talk) 00:21, 23 August 2024 (UTC)
- I think that the domain might have moved rather than being taken offline. That file is available at public
.dhe .ibm .com /software /globalization /gcoc /attachments /CP00850 .pdf. I'm not really sure what should be done here. McYeee (talk) 19:41, 23 August 2024 (UTC) - i think moving all www-03 references to the public.dhe domain should work. Notrealname1234 (talk) 20:34, 23 August 2024 (UTC)
- Note that it's not as simple as replacing www-03 with public.dhe. McYeee (talk) 20:38, 23 August 2024 (UTC)
- Thanks. I'll test for that soft-redirect rule, hunt for ghost redirects, filter for soft-404s, and crunchy-404s (WP:LINKROT#Glossary). IBM.com is notoriously complicated. -- GreenC 04:14, 24 August 2024 (UTC)
- Note that it's not as simple as replacing www-03 with public.dhe. McYeee (talk) 20:38, 23 August 2024 (UTC)
- i think moving all www-03 references to the public.dhe domain should work. Notrealname1234 (talk) 20:34, 23 August 2024 (UTC)
- I didn't find a good way to makes these live. The one method Notrealname1234 found worked for some of those PDF files ("systems_i_software_globalization_pdf"), not all. However that same method is good for ftp:// links noted in the next section below, because those links are not on the web (FTP protocol with no https access), and for that reason they have no archives available. Converting to https:// will be a big win. -- GreenC 20:02, 25 August 2024 (UTC)
- Enwiki Checked 724 pages and edited 642 pages. Moved 15 links to a new URL. Added 15
{{dead link}}. Switched 13|url-status=deadto live. Switched 78|url-status=liveto dead. Added 1,258 archive URLs (1,207 Wayback). - IABot DB - checked aprox 2,000 unique URLs. Changes will propagate to 300+ wikis.
Done -- GreenC 00:11, 26 August 2024 (UTC)
ftp:ftp.software.ibm.com
These can be replaced with https://public.dhe.ibm.com so long as the new URL is verified working.
- ftp://ftp.software.ibm.com/software/globalization/gcoc/attachments/CP01101.pdf --> https://public.dhe.ibm.com/software/globalization/gcoc/attachments/CP01101.pdf
120 pages. -- GreenC 19:04, 25 August 2024 (UTC)
- Enwiki: Checked 120 pages and edited 115 pages. Moved 206 links to a new URL. Removed 13
{{dead link}}. Added 69{{dead link}}. Switched 2|url-status=deadto live. Added 1 archive URLs (0 Wayback).
Done -- GreenC 01:52, 26 August 2024 (UTC)
articles.cnn.com
This is a mess of a domain where some things redirect and some things don't,,, I've found some patterns that work at least some of the time
More generally: http://articles.cnn.com/YYYY-MM-DD/EXT/WORDS.WITH.DOTS_1_WORDS-WITH-DASHES?_s=PM:THING
Goes to: https://www.cnn.com/YYYY/THING/MM/DD/WORDS.WITH.DOTS/index.html
words with dots and with years examples
|
|---|
|
http://articles.cnn.com/2001-09-16/us/inv.binladen.denial https://edition.cnn.com/2001/US/09/16/inv.binladen.denial/ |
More generally: http://articles.cnn.com/YYYY-MM-DD/ext/WORDS.WITH.DOTS
Goes to: https://edition.cnn.com/YYYY/EXT/MM/DD/WORDS.WITH.DOTS/
Similarly, you do the same thing if there's words with dashes (you can treat the URL as if it doesn't have anything after the _1_), such as in:
Those were the ones that I could find a somewhat consistent pattern for. Here's two where I couldn't quite, but I think somewhat of a pattern exists.
1467 pages GrapesRock (talk) 17:52, 26 August 2024 (UTC)
- In "example of words with dashes" .. I went ahead and programmed the rule, but the given example does not work ie.: https://edition.cnn.com/2011/WORLD/09/12/yemen.saleh.power.transfer/ .. hopefully the others will?
- I'll probably skip the Miscellany for now and see what is left when done the others. -- GreenC 16:44, 27 August 2024 (UTC)
- Hm, http://www.cnn.com/2011/WORLD/meast/09/12/yemen.saleh.power.transfer/index.html is where it is now which isn't helpful for generalization. Either I miscopied (a distinct possibility), or they changed up their domain again. Alas. GrapesRock (talk) 16:53, 27 August 2024 (UTC)
- It's OK the bot will try and skip any not working. Also finding some have a ghost redirect:
- -- GreenC 18:09, 27 August 2024 (UTC)
- Hm, http://www.cnn.com/2011/WORLD/meast/09/12/yemen.saleh.power.transfer/index.html is where it is now which isn't helpful for generalization. Either I miscopied (a distinct possibility), or they changed up their domain again. Alas. GrapesRock (talk) 16:53, 27 August 2024 (UTC)
- GrapesRock, to call this "done" is not accurate because there is probably more that could be done by searching and evaluation. Nevertheless, I'm going to mark it done for now and move on to other projects. If you discover other rules, I can undue the done tag and keep going. This is as you said initially a messy domain, like water from a stone, the "easy" ones are fixed and what remains is pretty difficult. -- GreenC 16:10, 28 August 2024 (UTC)
- Enwiki - Checked 1,469 pages and edited 557 pages. Moved 359 links to a new URL. Resolved 112 ghost redirects. Resolved 1 soft-404s. Removed 4
{{dead link}}. Added 24{{dead link}}. Switched 245|url-status=deadto live. Switched 20|url-status=liveto dead. Added 198 archive URLs (117 Wayback). Changed 5 citation metadata.
Done -- GreenC 16:10, 28 August 2024 (UTC)
cbsnews.com/stories
Hello. CBS News links with /stories/ in the URL don't work. www.cbsnews.com/stories/ is now www.cbsnews.com/news/name-of-the-article/. Some of these can be converted over while others don't fit the format: For example, this is now here for Pedro Carmona.
- Any punctuation marks in the article title are removed. For example, this is now here for Darleen Druyun. Same with this is now here with Kamal Derwish.
- However, this URL change doesn't always work as some also require the date at the end of the URL (day month year). For instance, this is now here for Pittsburgh Tribune-Review.
In this case, I think this changeover would need 3 stages. Article title, article title and date, archive any that remain broken.
Thanks! MrLinkinPark333 (talk) 23:28, 26 August 2024 (UTC)
- Building a new URL from
|title=data is difficult. In the above examples:- "CBS: Venezuelan Coup Leader Exits" --> "venezuelan-coup-leader-exits" (drop leading "CBS:")
- "Cashing In For Profit?" --> "cashing-in-for-profit" (drop ?)
- "'Lackawanna 6' Link To Yemen Killings?" --> "lackawanna-6-link-to-yemen-killings-04-11-2002" (drop single-quote and ?, add a date string parsed from original URL)
- "U.S. Plants: Open To Terrorists" --> "us-plants-open-to-terrorists-13-11-2003" (drop period and semi-colon, add a date string)
- In #1 and #4 they each have colons but are done differently. I suspect there will be a lot of edge cases. I can try some generic rules like this and see how many it can get. If you find any more rules, that will help. -- GreenC 03:20, 27 August 2024 (UTC)
- Other cases:
- The subtitle was removed from this and now that at Surveillance.
- Dollar sign was also removed from this to make that at Al Waleed bin Talal Al Saud.
- This is now here for Big Mac Index (no punctuation)
- Other cases:
- It might be easier to do ones without punctuation marks first. However, I can't predict which would need dates and which don't. MrLinkinPark333 (talk) 03:59, 27 August 2024 (UTC)
- "We're Watching: How Chicago Authorities Keep An Eye On The City" --> "were-watching" (drop punct and split : to left side)
- "$10 Million? NYC Says No Thanks" --> "10-million-nyc-says-no-thanks" (drop punct including $)
- "Iceland Says Bye to the Big Mac" --> "iceland-says-bye-to-the-big-mac" (square-link title)
- -- GreenC 15:40, 27 August 2024 (UTC)
- It might be easier to do ones without punctuation marks first. However, I can't predict which would need dates and which don't. MrLinkinPark333 (talk) 03:59, 27 August 2024 (UTC)
Results
- URLs with a match: 3,984 (converted via above method)
- URLs not matched: 1,293 (unable to covert)
- Title unspecified: 78 (bare and square links without a title)
User:MrLinkinPark333: This turned out better than expected with a 74% success rate. Though the 80/20 Rule is expected. It was fiddly getting all the transforms right and building a table of possible URLs. Some of the #2's probably have a match but the title is too complicated to parse. Many of the titles in #2 are straightforward but no URL exists. If you want the list of #2 let me know. -- GreenC 17:26, 29 August 2024 (UTC)
- If you mean the ones that were too complicated to convert, sure. Perhaps I can find more conversion rules from them. I'm also interested in the bare links of #3. I don't need the 404s of easy conversion. MrLinkinPark333 (talk) 17:44, 29 August 2024 (UTC)
- Set #2 and #3: Wikipedia:Link_rot/Cases/cbsnews.com-stories -- GreenC 05:47, 31 August 2024 (UTC)
- Of the 10 I tested in case #2, I found a handful that worked.
- Clam sauce this is now here after The Early Show - CBS News is removed
- Comparison of the healthcare systems in Canada and the United States, this is now here (subtitle is removed)
- Cold case - this this is now here - Everything after the slash is removed.
- Continuity of government: - this is now here (single quotes are removed)
- I would like the list for #2 updated at that link rot cases page if any more links are resolved. However, not all of these links will be fixed. For example, the links at Columbus Blue Jackets and Concerns and controversies at the 2010 Commonwealth Games don't have working links. If I find any more, I'll let you know. Otherwise, if we run out of ones to replace, the rest could be replaced with archived links. MrLinkinPark333 (talk) 20:07, 31 August 2024 (UTC)
- They all had archives added already there's no loss to verifiability if nothing further is done. The one's that might be made to work require special edge case rules that I don't want to deal with sorry it's too messy and time consuming there is too much variability. For example how many URLs are fixed by removing "The Early Show - CBS News"? The answer is 4. So that's 4 out of 1000. Cntrl-F search on "/" in that list, there is no general rule for "everything after slash is removed". It goes on like that, the data is extremely messy and variable. In situation like this, the 80/20 Rule rules - you can often get the first "easy" 80% and the remaining hard "20%" is dealt with or not, but at least you got 80% is better than nothing. It's just the nature of this particular problem trying to create a URL from free-form text. -- GreenC 23:23, 31 August 2024 (UTC)
- Fair enough. Hopefully there'd be more luck with the other cbs ones below. --MrLinkinPark333 (talk) 23:26, 31 August 2024 (UTC)
- Honestly, 74% is much better than I expected, considering. And probably at least half those in #2 are legitimate dead links no page available, the real conversion rate might be closer to 90%, after the dead links are factored out. -- GreenC 00:22, 1 September 2024 (UTC)
- Fair enough. Hopefully there'd be more luck with the other cbs ones below. --MrLinkinPark333 (talk) 23:26, 31 August 2024 (UTC)
- They all had archives added already there's no loss to verifiability if nothing further is done. The one's that might be made to work require special edge case rules that I don't want to deal with sorry it's too messy and time consuming there is too much variability. For example how many URLs are fixed by removing "The Early Show - CBS News"? The answer is 4. So that's 4 out of 1000. Cntrl-F search on "/" in that list, there is no general rule for "everything after slash is removed". It goes on like that, the data is extremely messy and variable. In situation like this, the 80/20 Rule rules - you can often get the first "easy" 80% and the remaining hard "20%" is dealt with or not, but at least you got 80% is better than nothing. It's just the nature of this particular problem trying to create a URL from free-form text. -- GreenC 23:23, 31 August 2024 (UTC)
- Of the 10 I tested in case #2, I found a handful that worked.
- Set #2 and #3: Wikipedia:Link_rot/Cases/cbsnews.com-stories -- GreenC 05:47, 31 August 2024 (UTC)
Done
cbsnews.com/numeric
Hello. While looking at CBS News, I found many URLs with numeric IDs that don't work. I found 2 that redirect but the rest don't:
- this redirects here for Judy Sheindlin.
- this redirects to here for 2020 United States presidential election in Idaho.
URL replacements are the same as the above section with some exceptions:
- For Jihobbyist, this is now here. - Political Hotshot needs to be removed from the reference as it does not exist in the new URL's article title.
- ~590 URLs that start with 2 (any non-mainspace can be ignored).
- ~4500 URLs that start with 8 (any non-mainspace can also be ignored)
Thanks again! MrLinkinPark333 (talk) 23:47, 26 August 2024 (UTC)
Results
- URLs with a match: 2,279 (converted with above method)
- URLs unable to match: 631
- URLs no title available: 36
Successful matches: 77.4% .. of those 631, roughly half are not a matching problem rather page no longer exists. Assuming 50% is true, and also removing the no title available, the real match rate is 88% ie. a further 12% might be matched but not practical to the variability of the data. -- GreenC 00:34, 1 September 2024 (UTC)
Done -- GreenC 00:34, 1 September 2024 (UTC)
- Not too bad! MrLinkinPark333 (talk) 03:44, 1 September 2024 (UTC)
cbc.ca/story
Hello. There are links to cbc.ca using /story/ that are broken. While there are new working URLS, I can't predict them. For example, this has a working archived link that redirects here for Scouting controversy and conflict. For these ones, I request looking for archived redirects first, then adding archives to the rest.
Thanks! MrLinkinPark333 (talk) 22:08, 28 August 2024 (UTC)
- This is a weird site because the ghost redirects are.. ghostly. In the above example, there are different redirects depending on timestamp. Sometimes it goes here and other times here. They are also somewhat chronologically buried in the list, normally I only get the most recent redirect (because there is no way of knowing which is correct without looking), and the last redirect goes here, which is not a ghost redirect. Thus unable to determine redirect URLs with automation. -- GreenC 21:06, 1 September 2024 (UTC)
- Checked 457 pages and edited 376 pages. Added 2
{{dead link}}. Switched 45|url-status=liveto dead. Added 384 archive URLs (337 Wayback). Changed 28 citation metadata.
Done -- GreenC 17:17, 3 September 2024 (UTC)
MrLinkinPark333: The cbc.ca/story and /news/story appear to have been parsed and fixed during the below section. Example Special:Diff/1243671672/1243757966 -- GreenC 17:17, 3 September 2024 (UTC)
www.cbc.ca/redirects
Hello. Cbc.ca has redirects to working URLs. Some of them require URL changes while others can be fixed quickly.
- No Changes
- This automatically goes here without any URL changes for Serena Williams. Not sure how many cases don't require /news/ to make working redirects.
- Changes
- 1a) Any with /m/ might need to be removed to make a redirect. For example, changing this to that points here for Commonwealth Games.
- 1b) However, that doesn't always work like this for House of Commons of Canada. Any of these or others that don't work would need archives, as I can't predict the new URL, which is this in the example.
- 2) Most URLs need to add /news/ to create a working redirect. For example, changing this into that makes the working URL for I'll Be Lovin' U Long Time.
- 3) However, an extra step for 2) might be needed if that doesn't work. Changing this to that makes a broken redirect. This redirect would need most of the URL changed to /amp/ to make the correct URL for Drake (musician).
- ~2000 2 folders
- ~3000 insource:/http?:\/\/www\.cbc\.ca\/[a-z]+\/[a-z]+\/[story]+\//
- ~1300 insource:/http?:\/\/www\.cbc\.ca\/[a-z]+\/[a-z]+\/[a-z]+\/[story]+\//
- ~50 insource:/http?:\/\/www\.cbc\.ca\/[a-z]+\/[a-z]+\/[a-z]+\/[a-z]+\/[story]+\//
- 3 insource:/http?:\/\/www\.cbc\.ca\/[a-z]+\/[a-z]+\/[a-z]+\/[a-z]+\/[a-z]+\/[story]+\//
Since this is a big request, I suggest focusing on ones that already redirect without changing the URLs first, then the ~180 /m/ ones. Thank you very much! MrLinkinPark333 (talk) 22:44, 28 August 2024 (UTC)
- Able to resolve 1b) by using the same technique in 3) ie. any url that ends in something like "1.2689859" can be tested for https://www.cbc.ca/amp/1.2689859
- URL like this can be changed to this (remove "/canada/" and change to "html")
- URL like this can be changed to this (add "/canada/") -- GreenC 04:52, 2 September 2024 (UTC)
- I tested this with the /news/story/ ones archived in the above section. This makes a working redirect for Vancouver and this makes a working redirect for Calgary Flames. Could these ones could be rechecked? Haven't figured out if the /news/ one have working redirects. MrLinkinPark333 (talk) 16:26, 2 September 2024 (UTC)
- Update: Got a few working redirect from /news/ with some changes. This works for Stan Lee. However, This needs to be changed to this to make a redirect for The Da Vinci Code. Same with this to that for Karla Homolka.MrLinkinPark333 (talk) 16:41, 2 September 2024 (UTC)
- This is getting complicated enough might need to run and see what's remaining for a second pass, otherwise could create rules contradictions. It's still processing but out of over 4000 edits so far there are about 300 it missed using the rules from yesterday. Stil need to check for soft404s and other problems. -- GreenC 16:49, 2 September 2024 (UTC)
- No worries! Those remaining rechecks could wait :) MrLinkinPark333 (talk) 17:06, 2 September 2024 (UTC)
- This is getting complicated enough might need to run and see what's remaining for a second pass, otherwise could create rules contradictions. It's still processing but out of over 4000 edits so far there are about 300 it missed using the rules from yesterday. Stil need to check for soft404s and other problems. -- GreenC 16:49, 2 September 2024 (UTC)
- Update: Got a few working redirect from /news/ with some changes. This works for Stan Lee. However, This needs to be changed to this to make a redirect for The Da Vinci Code. Same with this to that for Karla Homolka.MrLinkinPark333 (talk) 16:41, 2 September 2024 (UTC)
- I tested this with the /news/story/ ones archived in the above section. This makes a working redirect for Vancouver and this makes a working redirect for Calgary Flames. Could these ones could be rechecked? Haven't figured out if the /news/ one have working redirects. MrLinkinPark333 (talk) 16:26, 2 September 2024 (UTC)
- As the House of Commons of Canada example has both an /amp/ and full link, could those /amp/ ones be archived in case they break? Not sure why there's two links to the same article, but that helps! MrLinkinPark333 (talk) 16:09, 2 September 2024 (UTC)
- For House of Commons-like URLs I can't automatically determine the desktop URL only the mobile version. "AMP" is for pages optimized for mobile users, a parallel version of the site. Some sites have an API (a URL) that allows translation between the mobile and desktop URL ie. give it the AMP URL and it will return the desktop URL. Ideally all URLs on Wikipedia are the desktop version. But I don't know if they have an API, that would be nice to have. Either way anything added to Wikipedia will get archived into the Wayback Machine automatically. If the link later dies the bots or my tool will add an archive. -- GreenC 16:20, 2 September 2024 (UTC)
Results
- 10,368 links are live. All (but 31) are new, created per above rules.
- 95 links are not working. Of those, 12 had a
{{dead link}}added. The rest have archives. - Checked ____ pages and edited 7,133 pages. Moved 10,368 links to a new URL. Removed 45
{{dead link}}. Added 12{{dead link}}. Switched 1,460|url-status=deadto live. Switched 2|url-status=liveto dead. Added 556 archive URLs (402 Wayback).
Done -- GreenC 02:08, 3 September 2024 (UTC)
magxone.com
This website is dead. The current website gives a virus warning on my computer. Kaltenmeyer (talk) 03:50, 29 August 2024 (UTC)
Done: WP:JUDI batch #17 -- GreenC 04:31, 19 September 2024 (UTC)
ehdenfamilytree.com
This 'ehdenfamilytree.com' is dead and the new one is 'ehdenfamilytree.org'. Saroufim1 (talk) 01:49, 31 August 2024 (UTC)
- Checked 79 pages and edited 79 pages. Moved 120 links to a new URL. Removed 3
{{dead link}}. Switched 1|url-status=deadto live.
Done -- GreenC 19:26, 3 September 2024 (UTC)
www.lindsaygibsonpsyd.com
The official website listed for Lindsay Gibson redirects to watermillrestaurant.com, which appears to be the website for an Indonesian casino. I couldn't find an official dedicated website for Lindsay Gibson, and I'm not sure one currently exists. The best I could find were author pages on various other websites, none of which seem to serve as an official site. BlueEditorials (talk) 03:18, 31 August 2024 (UTC)
Done: WP:JUDI batch #17 -- GreenC 04:31, 19 September 2024 (UTC)
AnandTech shuts down
Amazing website/technews site AnandTech has shut down (https://www.anandtech.com/)
If an archive bot could preemptively archive the entirety of that website, that would be mint, as people are unsure what will happen to the content.
Thanks.
Headbomb {t · c · p · b} 04:32, 31 August 2024 (UTC)
1,158 pages — Preceding unsigned comment added by GreenC (talk • contribs)
- @GreenC: those are just what's used on Wikipedia. Which, I agree should be a priority. But if archiving the entirety of Andandtech is possible... either by talking to IA or through your bot or whatever that would be an amazing service to the tech community/tech historians. Headbomb {t · c · p · b} 14:45, 31 August 2024 (UTC)
- I believe the domain is already crawled by the Wayback Machine as part of the GDELT Collection ("NO404-GDELT"). For example given this archive the "About this capture" tab says GDELT Collection. The crawl was started in 2014, though it might be the whole site. If you can find some older URLs (older the better) and check if they exist in the Wayback. They should be there, but worth checking to see if the crawl missed them. If there are blank spots then I'll need to go through the URLs on Wikipedia one by one and capture any that are missing which is a bit of a job. -- GreenC 16:24, 31 August 2024 (UTC)
- Here's an article from 1998. You can tell it's a very early article by the URL: /161/ .. recent articles are at around /21000/. It's a pretty good bet the site is well archived. -- GreenC 05:29, 2 September 2024 (UTC)
- Sounds like anandtech.com will be staying stable and keeping all its articles up. [11].
And while the AnandTech staff is riding off into the sunset, I am happy to report that the site itself won’t be going anywhere for a while. Our publisher, Future PLC, will be keeping the AnandTech website and its many articles live indefinitely. So that all of the content we’ve created over the years remains accessible and citable.
Just FYI to help with making the decision. –Novem Linguae (talk) 17:41, 1 September 2024 (UTC)- There is a dedicated team of volunteers and staff (I supposed) of the Internet Archive archiving dead or dying websites. And Anandtech is listed on their wiki. If anyone here wants to speed up the process of the site getting archived, I suggest volunteering some time or resources there as well. – robertsky (talk) 05:52, 2 September 2024 (UTC)
- ω Awaiting to see if the site goes offline. -- GreenC 18:15, 3 September 2024 (UTC)
google.com/search?q=cache:
Practically all Google Search links with this string are redirects to Google cache, which has shut down. (technically, not every link starting with this string necessarily redirects to cache, but all links I've found are redirects). Helpful Raccoon (talk) 18:45, 1 September 2024 (UTC)
- Note: while the vast majority of URLs I found are followed by 12 characters and another colon before the original website URL (e.g. http://google.com/search?q=cache:EdF1mH2UVF8J:www.maurinet.com/allform/pportnew.pdf+mauritius+national+card&hl=en&ct=clnk&cd=5&gl=nz in Identity document), a few of them are not followed by 12 characters (e.g. http://www.google.com/search?q=cache:www.melafoundation.org/theatre.pdf in Drone music). Helpful Raccoon (talk) 18:58, 1 September 2024 (UTC)
- OK. I wrote/use Google Cache Parser (GitHub). It correctly parses both those URLs. -- GreenC 21:53, 4 September 2024 (UTC)
User:Helpful Raccoon: I cleared Google Cache in February: Wikipedia:Link_rot/URL_change_requests/Archives/2024/February#Google_cache targeting webcache.googleusercontent.com but was not aware of google.com/search?q=cache: .. thanks for bringing this to attention. 776 pages-- GreenC 19:08, 1 September 2024 (UTC)
Results
- Converted 810 URLs
Done -- GreenC 23:40, 4 September 2024 (UTC)
time-blog.com
Site appears to be dead. All links redirect to the time.com homepage. There are 54 pages. Thank you! Helpful Raccoon (talk) 23:13, 1 September 2024 (UTC)
Enwiki
- Checked 54 pages and edited 41 pages. Added 2
{{dead link}}. Switched 4|url-status=liveto dead. Added 44 archive URLs (43 Wayback).
IABot DB
- Checked and updated 84 unique URLs which will propagate across 300+ wikis.
Done -- GreenC 00:29, 5 September 2024 (UTC)
nola.com/politics
This subpage appears to be dead. All links currently redirect to https://www.theadvocate.com/baton_rouge/news/politics/ (and it doesn't show the original article). I could not find the original articles by searching in theadvocate.com. There are 343 pages. Helpful Raccoon (talk) 23:23, 1 September 2024 (UTC)
Enwiki
- Checked 342 pages and edited 320 pages. Added 30
{{dead link}}. Switched 36|url-status=liveto dead. Added 563 archive URLs (419 Wayback). Changed 22 citation metadata.
IABot DB
- Checked and updated 677 unique links which will propagate to 300+ wikis
Done -- GreenC 05:03, 5 September 2024 (UTC)
voices.washingtonpost.com
Articles appear to be unavailable. All links currently redirect to the Washington Post landing page; many of them already have archive URLs but a significant minority most do not. 1874 pages. Helpful Raccoon (talk) 23:36, 1 September 2024 (UTC)
Enwiki
- Checked 1,876 pages and edited 1,758 pages. Added 26
{{dead link}}. Switched 294|url-status=liveto dead. Added 2,050 archive URLs (1,750 Wayback). Changed 56 citation metadata.
IABot DB
- Checked and updated about 2,500 unique links which will propagate to 300+ wikis
Done -- GreenC 01:36, 6 September 2024 (UTC)
articles.nydailynews.com
This site is down, but links can be converted to live subpages of nydailynews.com if the article title is known: Currently, links are of the form articles.nydailynews.com/[yyyy]-[mm]-[dd]/[section]/[junk]. These articles are available at URLs of the form www.nydailynews.com/[yyyy]/[mm]/[dd]/[title]/, where the title is in all lowercase, punctuation is stripped, and spaces are replaced by hyphens. Not sure about edge cases. Note that the hyphens in the dates must be replaced with slashes.
The title can be extracted if an archived version of the articles.nydailynews.com article is available; alternatively, |title= data can be used, but it does not always correspond to the actual article title due to human error. Here is an example of an archived page. Thank you! 1381 pages. Helpful Raccoon (talk) 00:30, 2 September 2024 (UTC)
- Turns out that the article dates sometimes change around too... (8/13 vs 8/14 in the following example)
- Old article example: http://articles.nydailynews.com/2012-08-14/news/33187461_1_giants-weatherford-giants-and-jets-metlife-stadium (from New York Giants)
- New article example: https://www.nydailynews.com/2012/08/13/weatherford-beating-jets-is-pretty-sweet/ Helpful Raccoon (talk) 02:09, 2 September 2024 (UTC)
- Some articles also might just be lost; e.g. I couldn't find a live version of https://web.archive.org/web/20121028212158/http://articles.nydailynews.com/2012-05-15/news/31714265_1_john-mayer-spotlight-interviews. Helpful Raccoon (talk) 02:13, 2 September 2024 (UTC)
User:Helpful Raccoon: I can do this (with a ~10 to 20% miss rate), but, adding archives may be better than converting to live because the live appears to be paywalled (example). Granted the paywall is "low" ie. one can view source to read the content; Or save the page at Wayback which removes the paywall (example) .. let me know what you think. My estimate is treat them all as dead and add archives. -- GreenC 05:15, 2 September 2024 (UTC)
- I don't have a preference honestly. Archiving is at least a simpler solution than converting to live URLs. Helpful Raccoon (talk) 05:26, 2 September 2024 (UTC)
- I can do it, only what is best for Wikipedia. The URLs are almost identical to Wikipedia:Link_rot/URL_change_requests#articles.cnn.com and the conversion of title to URL is basically the same as Wikipedia:Link_rot/URL_change_requests#cbsnews.com/stories. In the past, I usually lean towards archives over live when there is a paywall, to make verification easier. Sometimes website will deny archive access, at which point the URLs become inaccessible (dead at the site and no archives), until they are converted to the live version. Pros and cons, reactive and proactive. -- GreenC 05:59, 2 September 2024 (UTC)
- Due to the paywall I simply converted them to archives. If the situation changes I can redo to the live link method. -- GreenC 16:45, 6 September 2024 (UTC)
- I can do it, only what is best for Wikipedia. The URLs are almost identical to Wikipedia:Link_rot/URL_change_requests#articles.cnn.com and the conversion of title to URL is basically the same as Wikipedia:Link_rot/URL_change_requests#cbsnews.com/stories. In the past, I usually lean towards archives over live when there is a paywall, to make verification easier. Sometimes website will deny archive access, at which point the URLs become inaccessible (dead at the site and no archives), until they are converted to the live version. Pros and cons, reactive and proactive. -- GreenC 05:59, 2 September 2024 (UTC)
Enwiki
- Checked 1,380 pages and edited 1,072 pages. Added 250
{{dead link}}. Switched 146|url-status=liveto dead. Added 920 archive URLs (368 Wayback).
IABot DB
- Checked and updated about 2,000 unique links which will propagate across 300+ wikis
Done -- GreenC 23:05, 6 September 2024 (UTC)
weeklystandard.com
Dead website for a defunct magazine, The Weekly Standard. 989 pages. Helpful Raccoon (talk) 04:48, 2 September 2024 (UTC)
Enwiki
- Checked 990 pages and edited 744 pages. Added 31
{{dead link}}. Switched 141|url-status=liveto dead. Added 707 archive URLs (618 Wayback). Changed 40 citation metadata.
IABot DB
- Checked 1,591 and updated links which will propagate to 300+ wikis
Done --GreenC 02:42, 7 September 2024 (UTC)
archive.fortune.com
This site is dead and no articles are available on fortune.com, but they are currently available on CNN for some reason. "archive.fortune.com" just needs to be replaced with "money.cnn.com" in all URLs. Example dead URL from Apple Inc.: http://archive.fortune.com/magazines/fortune/fortune_archive/2007/03/19/8402321/index.htm. Equivalent live URL: https://money.cnn.com/magazines/fortune/fortune_archive/2007/03/19/8402321/index.htm. 569 pages. Helpful Raccoon (talk) 05:22, 2 September 2024 (UTC)
Enwiki
- Checked 569 pages and edited 561 pages. Moved 616 links to a new URL. Removed 3
{{dead link}}. Added 2{{dead link}}. Switched 49|url-status=deadto live. Switched 2|url-status=liveto dead. Added 16 archive URLs (10 Wayback). Changed 1 citation metadata.
IABot DB
- Checked and updated 799 links which will propagate to 300+ wikis
Done -- GreenC 02:45, 7 September 2024 (UTC)
- GreenC (talk · contribs), this edit changed the URL to https://www.cnn.com/business which seems to be incorrect. Would you take a look? Thank you. Cunard (talk) 08:28, 7 September 2024 (UTC)
- I found 6 other edits that added the cnn.com/business URL (plus one that was made by a different user): [12] [13] [14] [15] [16] [17]. Helpful Raccoon (talk) 09:40, 7 September 2024 (UTC)
- That's a soft-404, and can clearly see in the logs. Trying to do too much and skipping steps. I'll roll back those edits and redo the pages, with the soft-404 trap enabled. There are no others (that I can see in the logs). Thanks for the notification. -- GreenC 15:56, 7 September 2024 (UTC)
- Fixed eg. [18] -- GreenC 19:20, 7 September 2024 (UTC)
articles.chicagotribune.com
Articles in this domain currently redirect to a 404 page. However, most can be converted into live URLs using the same method for articles.nydailynews.com. Example dead URL from Barack Obama: http://articles.chicagotribune.com/2009-03-22/features/0903200725_1_barack-obama-story-chicago-school-harvard-law. Converted live URL: https://www.chicagotribune.com/2009/03/22/ivory-tower-of-power/. 12,469 pages. Helpful Raccoon (talk) 18:46, 2 September 2024 (UTC)
- When doing free-form title string conversions such as this, expect 10% to 20% won't convert for various reasons, mainly because of variable title strings don't match, or the page is legitimately no longer available at the site. -- GreenC 17:16, 8 September 2024 (UTC)
Enwiki in two batches:
- Batch 1: Checked 3,000 pages and edited 2,937 pages. Moved 3,558 links to a new URL. Resolved 336 ghost redirects. Resolved 44 soft-404s. Removed 2
{{dead link}}. Added 11{{dead link}}. Switched 202|url-status=deadto live. Switched 27|url-status=liveto dead. Added 446 archive URLs (383 Wayback). Changed 36 citation metadata. - Batch 2: Checked 9,500 pages and edited 9,269 pages. Moved 11,141 links to a new URL. Resolved 818 ghost redirects. Resolved 115 soft-404s. Removed 3
{{dead link}}. Added 63{{dead link}}. Switched 639|url-status=deadto live. Switched 122|url-status=liveto dead. Added 1,418 archive URLs (1,301 Wayback). Changed 122 citation metadata.
IABot DB:
- Checked and updated 18,045 unique links which will propagate to 300+ wikis
Done -- GreenC 23:43, 10 September 2024 (UTC)
sportsillustrated.cnn.com
Dead domain, along with subdomains such as vault.sportsillustrated.cnn.com. I could not find any live versions of the articles. 9,555 pages. Helpful Raccoon (talk) 19:16, 2 September 2024 (UTC)
- Wait, some articles are live on si.com. I'm working on possible conversion rules. Helpful Raccoon (talk) 04:51, 7 September 2024 (UTC)
- What I have so far: Articles may be live if they are in the vault (vault.sportsillustrated.cnn.com or sportsillustrated.cnn.com/vault) or were published after 2008 or so.
- New vault URLs are of the form https://vault.si.com/vault/YYYY/MM/DD/name-of-article. Unfortunately the original URL does not contain the date of publication, but this can be extracted from the reference template or an archived version of the original article. Original article example from Baseball: http://sportsillustrated.cnn.com/vault/article/magazine/MAG1188950/index.htm. Converted: https://vault.si.com/vault/2011/08/08/its-all-about-anticipation.
- Articles published around 2013-2014 are typically of the form sportsillustrated.cnn.com/<section>/news/YYYYMMDD/name-of-article/.... The converted article is of the form si.com/<new section>/YYYY/MM/DD/name-of-article. ".ap" and any other junk should be stripped from the end of the original URL. The date sometimes changes by 1 day. Old URL example from Condoleezza Rice: http://sportsillustrated.cnn.com/college-football/news/20131016/condoleezza-rice-college-football-playoff/index.html. New URL: https://www.si.com/college/2013/10/17/condoleezza-rice-college-football-playoff.
- In many cases the section is unchanged during the conversion, but there are some special cases. Section conversion rules that I've found: college-football to college, college-basketball to basketball, -olympics to olympics.
- Articles published between 2009-2013 are usually live, but the conversion rules can be difficult. I will get to those later. Helpful Raccoon (talk) 05:29, 7 September 2024 (UTC)
- Hi, I appreciate the discoveries you made. The above is a programmers nightmare: "around 2013-2014 are typically" etc.. etc.. I have limits, this is one. The above is probably 30-50 hours (3-5 days) given the number of links, and the likely amount of novel code and testing involved. It seems easy, but is not. It's not in my budget sorry. In the mean time I can convert to archives, and if someone wants to do these conversions, send me the table of old and new I will add them to wiki with appropriate template support. -- GreenC 06:28, 7 September 2024 (UTC)
- Thanks for the feedback. I will try to be more precise when requesting conversions and take into account potential difficulties. Helpful Raccoon (talk) 09:04, 7 September 2024 (UTC)
- Hi, I appreciate the discoveries you made. The above is a programmers nightmare: "around 2013-2014 are typically" etc.. etc.. I have limits, this is one. The above is probably 30-50 hours (3-5 days) given the number of links, and the likely amount of novel code and testing involved. It seems easy, but is not. It's not in my budget sorry. In the mean time I can convert to archives, and if someone wants to do these conversions, send me the table of old and new I will add them to wiki with appropriate template support. -- GreenC 06:28, 7 September 2024 (UTC)
Enwiki
- Checked 9,562 pages and edited 7,024 pages. Removed 3
{{dead link}}. Added 237{{dead link}}. Switched 1,079|url-status=liveto dead. Added 8,136 archive URLs (6,462 Wayback). Changed 5 citation metadata.
IABot DB
- Checked and updated about 15,000 unique links which will propagate to 300+ wikis
Done -- GreenC 15:03, 8 September 2024 (UTC)
blogs.cnn.com
Defunct domain that used various subdomains, such as news.blogs.cnn.com and thechart.blogs.cnn.com. These subdomains all give 410 errors. Most articles do not appear to be live at the main cnn.com domain, although I did find one: http://geekout.blogs.cnn.com/2012/04/11/stan-lee-launches-his-own-comic-convention/ from Stan Lee is live at https://www.cnn.com/2012/04/11/living/stan-lee-launches-his-own-comic-convention. It might be best to just mark all as dead. 2,524 pages. Helpful Raccoon (talk) 19:32, 2 September 2024 (UTC)
Enwiki
- Checked 2,527 pages and edited 1,665 pages. Added 47
{{dead link}}. Switched 442|url-status=liveto dead. Added 1,660 archive URLs (1,353 Wayback). Changed 5 citation metadata.
IABot DB
- Checked and updated about 4,000 unique URLs which will propagate to over 300+ wikis
Done -- GreenC 22:51, 11 September 2024 (UTC)
Bug: square archives
There was a bug in the core code, introduced 22 projects ago. All archive URLs with a square link were skipped. Thus something like [https://web.archive.org/web/20240101/https://example.com Example.com] was not processed. Following is the list of projects (internal code). I may or may not redo them as time allows.
- urlchanger_www03ibmcom.nim
- urlchanger_tsfi.nim
urlchanger_ieee.nim- urlchanger_ftpibmcom.nim
- urlchanger_fhwadotgov.nim
- urlchanger_wileycomstore.nim
- urlchanger_msnbcmsncom.nim
- urlchanger_hpvectorcojp.nim
- urlchanger_bcsportshalloffame.nim
- urlchanger_nbcnewscom.nim
- urlchanger_gameinformer.nim
urlchanger_cbccastory.nim- urlchanger_ukbusinessinsidercom.nim
- urlchanger_cbsnewsnumeric.nim
- urlchanger_slatemsncom.nim
- urlchanger_articlescnncom.nim
urlchanger_prwebcom.nimurlchanger_articleslatimescom.nim- urlchanger_emporis3.nim
- urlchanger_cbsnewsstories.nim
- urlchanger_cartoonnetwork.nim
- urlchanger_businessinsidercomau.nim
-- GreenC 22:04, 2 September 2024 (UTC)
- These are not big numbers. In the 10,333 URLs for cbc.ca/redirects above, there were 60 instances of square archives. I'll re-run a couple of the larger projects. -- GreenC 02:40, 3 September 2024 (UTC)
xyz.reuters.com
uk.reuters.com, ca.reuters.com: Some (but not all) subpages in these domains are soft redirects which ca be converted to live URLs by replacing the domain with just "reuters.com", no subdomain. E.g. http://uk.reuters.com/article/wtMostRead/idUKTRE50318U20090104 in Matt Smith can be converted to http://www.reuters.com/article/wtMostRead/idUKTRE50318U20090104.
However, this conversion often leads to an unrelated article for some reason. For example, the URL http://uk.reuters.com/article/idUKN1420378520061215 from Korn would get converted to http://reuters.com/article/idUKN1420378520061215, which is a different article. The original article in this case appears to be completely dead. Either the title at the converted URL needs to be extracted to see if the article is correct, or else no conversion should happen at all.
in.reuters.com: Some links are soft redirects which can be converted like the above. This occurs when the URL contains keywords before "idINIndia", e.g. http://in.reuters.com/article/film-treysongz-idINDEE9010B720130102 in Trey Songz. Other links are either completely dead or soft redirects with unpredictable conversion rules, e.g. http://in.reuters.com/article/idINIndia-54075420110111 in Ricky Ponting. In this case, there are live URLs, e.g. https://www.reuters.com/article/sports/ponting-should-focus-on-batting-wessels-idUSTRE70A2EU/, but I can't find a way to get the correct "id" at the end of the URL.
uk.reuters.com: 6,893 pages.
ca.reuters.com: 572 pages.
in.reuters.com: 2,588 pages.
Helpful Raccoon (talk) 01:29, 3 September 2024 (UTC)
- Due to the unreliability of the above simple conversion rules, I'd say archiving everything is best unless there's a feasible workaround. Helpful Raccoon (talk) 09:18, 7 September 2024 (UTC)
- I agree. It's also probably dangerous to convert these because in the future they might recycle IDs as they appear to have done already. Considering how long those IDs are, you'd think they would remain unique until the end of time, but they seem to be reused for different articles based on the host name. It's a yellow flag about their system. Might be a consequence of how the site grew geographically over time. -- GreenC 05:32, 8 September 2024 (UTC)
Enwiki
- Checked 9,575 pages and edited 8,473 pages. Converted 1 templates. Added 748
{{dead link}}. Switched 2,124|url-status=liveto dead. Added 7,922 archive URLs (6,815 Wayback). Changed 272 citation metadata.
IABot DB
- Checked 22,154 links and updated 15,124 which will propagate to 300+ wikis
Done -- GreenC 15:29, 13 September 2024 (UTC)
xroads.virginia.edu
A defunct project where all subpages return 404 errors; https://xroads.virginia.edu/ recommends using Wayback as one option. 562 pages. Helpful Raccoon (talk) 18:38, 3 September 2024 (UTC)
Enwiki
- Checked 576 pages and edited 385 pages. Added 5
{{dead link}}. Switched 24|url-status=liveto dead. Added 421 archive URLs (385 Wayback).
IABot DB
- Checked and updated 789 links which will propagate to 300+ wikis.
Done -- GreenC 18:01, 13 September 2024 (UTC)
au.af.mil
This domain and all subdomains are dead. Unable to find live versions of the sources. 692 pages. Helpful Raccoon (talk) 18:47, 3 September 2024 (UTC)
Enwiki
- Checked 693 pages and edited 437 pages. Added 26
{{dead link}}. Switched 38|url-status=liveto dead. Added 444 archive URLs (424 Wayback).
IABot DB
- Checked and updated 819 links which propagate to 300+ wikis
Done -- GreenC 00:51, 14 September 2024 (UTC)
arxiv.org mirror shut down
arxiv.org has several mirrors but these will be shut down on 2024-09-15
Only one mirror has a domain name that arxiv.org does not control x x x.lanl.gov from US Los Alamos National Laboratory (remove spaces between x's)
Can we get all links of the format https://x x x .lanl.org/{path} changed to https://arxiv.org/{path} ?
All links from cn.arxiv.org, de.arxiv.org lanl.arxiv.org and in.arxiv.org will be rerouted via DNS changes to arxiv.org and continue to work correctly. URLs with those hostnames could also be updated but it is unnecessary. Brian Caruso (talk) 14:37, 5 September 2024 (UTC)
- There's no xxx.lanl.gov link anywhere on Wikipedia ([19]). For cn. de. lanl. and in.arxiv, there's about 27 links ([20]), which I will shortly update. Headbomb {t · c · p · b} 15:01, 5 September 2024 (UTC)
theweeklystandard.com
Dead. 9 pages. -- GreenC 17:59, 7 September 2024 (UTC)
Already done -- GreenC 00:53, 14 September 2024 (UTC)
nhl.com/gamecenter
NHL Gamecenter changes their URLs sometimes in 2019–2020. The old addresses all have the format:
http://www.nhl.com/gamecenter/en/recap?id=
followed by a number.
If you use:
https://www.nhl.com/gamecenter/
followed by the number you get redirected to the new page.
As an example if the old URL is:
http://www.nhl.com/gamecenter/en/recap?id=2003030411
and you use the URL:
http://www.nhl.com/gamecenter/2003030411
you will be redirected to:
https://www.nhl.com/gamecenter/cgy-vs-tbl/2004/05/25/2003030411.
There appears to be a few hundred URLs that need updating. -- LCU ActivelyDisinterested «@» °∆t° 19:31, 7 September 2024 (UTC)
- I would expect that they will delete the redirects at some point, so it would be good to update to the new URL if that's possible. I don't know how difficult that will be given you have to call the redirect to get the final URL. -- LCU ActivelyDisinterested «@» °∆t° 19:37, 7 September 2024 (UTC)
- Not difficult! -- GreenC 05:04, 8 September 2024 (UTC)
- Brilliant thanks GreenC. -- LCU ActivelyDisinterested «@» °∆t° 09:17, 8 September 2024 (UTC)
- User:ActivelyDisinterested, here you go over 20,000 URLs changed in 682 pages (beginning upload now). -- GreenC 15:08, 14 September 2024 (UTC)
- I had not realised the scale, fantastic work! -- LCU ActivelyDisinterested «@» °∆t° 16:35, 14 September 2024 (UTC)
- The "season" pages have most of it. Like Special:Diff/1235871408/1245695508, where it changed "ott-vs-buf" -> "buf-vs-ott" - they redid in alphabetical order and thankfully created redirects. I'm also doing the IABot database, but IABot does not support URL moves, so unfortanately all these (without working redirects) will be considered dead links with archives added. It will only effect the non-Enwiki wikis. -- GreenC 18:07, 14 September 2024 (UTC)
- I had not realised the scale, fantastic work! -- LCU ActivelyDisinterested «@» °∆t° 16:35, 14 September 2024 (UTC)
- User:ActivelyDisinterested, here you go over 20,000 URLs changed in 682 pages (beginning upload now). -- GreenC 15:08, 14 September 2024 (UTC)
- Brilliant thanks GreenC. -- LCU ActivelyDisinterested «@» °∆t° 09:17, 8 September 2024 (UTC)
- Not difficult! -- GreenC 05:04, 8 September 2024 (UTC)
Enwiki
- Checked 980 pages and edited 682 pages. Moved 20,149 links to a new URL. Resolved 58 ghost redirects. Resolved 344 soft-404s. Removed 2
{{dead link}}. Added 33{{dead link}}. Switched 9|url-status=deadto live. Added 254 archive URLs (211 Wayback).
IABot DB
- Checked 21,348 links and updated 9,298 which propagate to 300+ wikis
Done -- GreenC 02:12, 15 September 2024 (UTC)
yemenileopard.org
Has been usurped by an advert for a mobile game Big Blue Cray(fish) Twins (talk) 22:38, 7 September 2024 (UTC)
Done: WP:JUDI batch #17 -- GreenC 04:30, 19 September 2024 (UTC)
pubmedcentral.nih.gov
These ([21]) links all seem dead or redirecting to a 404. replacing them however with the below seems to make them work:
http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1380757
to
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1380757/
Jonatan Svensson Glad (talk) 19:46, 8 September 2024 (UTC)
Enwiki
- Checked 180 pages and edited 180 pages. Moved 193 links to a new URL. Removed 2
{{dead link}}. Added 3{{dead link}}. Added 3 archive URLs (3 Wayback).
IABot DB
- Checked and updated 1,204 links which propagate to 300+ wikis
Done -- GreenC 14:24, 15 September 2024 (UTC)
Various articles.<domain>.com subdomains for Tribune Publishing sites
Other publications owned by Tribune Publishing besides New York Daily News and Chicago Tribune have defunct subdomains of the form articles.<domain>.com that can be transformed to live links in the same way as articles.nydailynews.com and articles.chicagotribune.com. Paywalled sites can be archived instead of converted.
articles.courant.com (No paywall, 1,594 pages)
articles.sun-sentinel.com (Paywall, 3,369 pages)
articles.dailypress.com (No paywall, 565 pages)
articles.orlandosentinel.com (No paywall, 3,544 pages)
articles.mcall.com (No paywall, 1,170 pages)
The Virginian-Pilot does not have any links of the form "articles.pilotonline.com", presumably because Tribune Publishing acquired it very recently. Helpful Raccoon (talk) 16:51, 10 September 2024 (UTC)
- User:Helpful_Raccoon, nice find. May I ask, would you redo the request so each has its own section? Thus 5 sections. It's because the edit summary links to a domain name, which I need to run one at a time. You can delete this comment. -- GreenC 19:46, 10 September 2024 (UTC)
- Will do! Helpful Raccoon (talk) 21:38, 10 September 2024 (UTC)
articles.courant.com
Should be converted in the same way as articles.nydailynews.com and articles.chicagotribune.com.
(No paywall, 1,594 pages) Helpful Raccoon (talk) 21:40, 10 September 2024 (UTC)
Enwiki
- Checked 1,596 pages and edited 1,547 pages. Moved 1,855 links to a new URL. Resolved 28 ghost redirects. Resolved 7 soft-404s. Removed 5
{{dead link}}. Added 11{{dead link}}. Switched 239|url-status=deadto live. Switched 17|url-status=liveto dead. Added 251 archive URLs (205 Wayback). Changed 19 citation metadata.
IABot DB
- Checked and updated 2,360 unique links which propagate to 300+ wikis
Done -- GreenC 00:10, 16 September 2024 (UTC)
articles.sun-sentinel.com
Can be converted in the same way as articles.nydailynews.com and articles.chicagotribune.com, but has a paywall, so archiving is probably best.
(Paywall, 3,369 pages) Helpful Raccoon (talk) 21:41, 10 September 2024 (UTC)
Enwiki
- Checked 3,370 pages and edited 2,292 pages. Added 59
{{dead link}}. Switched 392|url-status=liveto dead. Added 2,705 archive URLs (2,482 Wayback). Changed 24 citation metadata.
IABot DB
- Checked and updated 5,100 links which will propagate to 300+ wikis
Done -- GreenC 18:30, 16 September 2024 (UTC)
articles.dailypress.com
Should be converted in the same way as articles.nydailynews.com and articles.chicagotribune.com.
(No paywall, 565 pages) Helpful Raccoon (talk) 21:42, 10 September 2024 (UTC)
Enwiki
- Checked 564 pages and edited 538 pages. Moved 627 links to a new URL. Resolved 16 ghost redirects. Resolved 6 soft-404s. Removed 3
{{dead link}}. Added 6{{dead link}}. Switched 172|url-status=deadto live. Switched 2|url-status=liveto dead. Added 76 archive URLs (60 Wayback). Changed 8 citation metadata.
IABot DB:
- Checked and updated 783 unique links which will propagate to 300+ wikis
Done -- GreenC 20:36, 23 September 2024 (UTC)
articles.orlandosentinel.com
Should be converted in the same way as articles.nydailynews.com and articles.chicagotribune.com.
(No paywall, 3,544 pages) Helpful Raccoon (talk) 21:43, 10 September 2024 (UTC)
Enwiki
- Checked 3,541 pages and edited 3,431 pages. Moved 4,156 links to a new URL. Resolved 80 ghost redirects. Resolved 24 soft-404s. Removed 1
{{dead link}}. Added 67{{dead link}}. Switched 503|url-status=deadto live. Switched 49|url-status=liveto dead. Added 598 archive URLs (505 Wayback). Changed 24 citation metadata.
IABot DB
- Checked and updated about 6,000 unique links which will propagate to 300+ wikis
Done -- GreenC 16:04, 24 September 2024 (UTC)
articles.mcall.com
Should be converted in the same way as articles.nydailynews.com and articles.chicagotribune.com.
(No paywall, 1,170 pages) Helpful Raccoon (talk) 21:43, 10 September 2024 (UTC)
Enwiki
- Checked 1,170 pages and edited 1,148 pages. Moved 1,316 links to a new URL. Resolved 14 ghost redirects. Resolved 7 soft-404s. Added 7
{{dead link}}. Switched 176|url-status=deadto live. Switched 73|url-status=liveto dead. Added 180 archive URLs (121 Wayback). Changed 28 citation metadata.
IABot DB
- Checked and updated 1,722 unique links which propagate to 300+ wikis
Done -- GreenC 20:42, 24 September 2024 (UTC)
pubs.acs.org
These ~80 articles has link which are dead. There may be more if making better a better search query. I believe they can be replaced the following way:
- Example
From: http://pubs.acs.org/cgi-bin/abstract.cgi/jafcau/1999/47/i05/abs/jf981170m.html
To: https://pubs.acs.org/doi/10.1021/jf981170m
--Jonatan Svensson Glad (talk) 04:50, 11 September 2024 (UTC)
- They should use (1)
|doi=or (2) {{doi}} rarther than a URL to the publisher's website, or simply be (3) removed altogether if there already is a proper doi link. I just took care of about half of the (3) cases. DMacks (talk) 05:12, 11 September 2024 (UTC)- There are a bunch of variations...working on it manually... DMacks (talk) 13:45, 11 September 2024 (UTC)
- Mainspace all done. Zero of them were worthy of remaining as a URL at all:) DMacks (talk) 18:50, 11 September 2024 (UTC)
- There are a bunch of variations...working on it manually... DMacks (talk) 13:45, 11 September 2024 (UTC)
Already done thank you DMacks -- GreenC 00:34, 25 September 2024 (UTC)
timesonline.co.uk
Old URLs for The Times don't work. While some of these have new URLs at thetimes.com, they can't be easily converted . For example, this is now here for Adele. Unfortunately, I think all of these links and the subdomains (entertainment.timesonline.co.uk, business.timesonline.co.uk, etc.) will need archives. It might be easier to do the subdomains first. Some articles already have archived links added like at Premier League. 15,000+ articles altogether. Thank you! MrLinkinPark333 (talk) 19:34, 12 September 2024 (UTC)
This is a difficult project due to a large number of soft-404s within archives:
soft404 rules for archives
|
|---|
if url ~ "(the)?(sunday)?(times(plus|online)?)[.]co[.]uk":
if url ~ "login=false":
return "Check 6.131"
if url ~ "(the)?(sunday)?(times(plus|online)?)[.]co[.]uk/[st][to][lo]/[?]CMP=":
return "Check 6.132"
if url ~ "(the)?(sunday)?(times(plus|online)?)[.]co[.]uk/[st][to][lo]/news/?([?](token=null|id=[a-zA-Z0-9]{2,10}$))?":
return "Check 6.137"
if url ~ "(the)?(sunday)?(times(plus|online)?)[.]co[.]uk/[st][to][lo]/(news|news/world|tv-radio|business|travel|arts|arts/(film/reviews|tv-radio))/?$":
return "Check 6.135"
if url ~ "the-tls[.]co[.]uk/tls/?$":
return "Check 6.136"
gsubs("://", "__T__", url)
if url ~ "//":
return "Check 6.133"
gsubs("__T__", "://", url)
if url ~ "obituaries/?$":
return "Check 6.134"
|
..where "url" is the redirected URL the page was saved from, as indicated on the archive page ie. not the URL on wiki or the live redirect (if any).
Enwiki
- Checked 15,686 pages and edited 13,589 pages. Moved 275 links to a new URL. Resolved 20,115 soft-404s. Removed 4
{{dead link}}. Added 6,721{{dead link}}. Switched 28|url-status=deadto live. Switched 1,736|url-status=liveto dead. Added 8,624 archive URLs (7,156 Wayback). Changed 593 citation metadata.
- Explanation: the bot analyzed about 20,000 URLs - all dead and presenting as soft-404. Of those, about 17,000 the bot added an archive URL, dead link template or switched url-status to dead. The other 3,000 are uncertain but probably already have an archive URL and url-status=dead ie. nothing to do. The large number 6,721
{{dead link}}is unfortunate, it represents the problem noted above of archives containing soft-404. -- GreenC 19:21, 26 September 2024 (UTC)
- Explanation: the bot analyzed about 20,000 URLs - all dead and presenting as soft-404. Of those, about 17,000 the bot added an archive URL, dead link template or switched url-status to dead. The other 3,000 are uncertain but probably already have an archive URL and url-status=dead ie. nothing to do. The large number 6,721
- That's too bad with the large about of dead links. If the new URLs were easy to convert, we could have swapped them over. Thank you for working on this! MrLinkinPark333 (talk) 19:25, 26 September 2024 (UTC)
- Yeah this domain needed help because it was marked "Subscription" in the IABot DB (ie. skip processing), so most of them were dead with no archives. Normally I would "done" at this point, but I want to try a new experimental method for finding the live URL (it has a low probability of success) - I won't be able to start until next week. -- GreenC 13:23, 27 September 2024 (UTC)
- Experimental method not working. -- GreenC 16:25, 30 September 2024 (UTC)
- Yeah this domain needed help because it was marked "Subscription" in the IABot DB (ie. skip processing), so most of them were dead with no archives. Normally I would "done" at this point, but I want to try a new experimental method for finding the live URL (it has a low probability of success) - I won't be able to start until next week. -- GreenC 13:23, 27 September 2024 (UTC)
IABot DB
- Checked and edited about 28,000 links which will propagate to 300+ wikis
Done -- GreenC 16:25, 30 September 2024 (UTC)
foxnews.com/story
Old URLs for foxnews.com with numeric IDs either redirect to new URLs, redirect to the wrong page or are broken. Working URLs are mainly at www.foxnews.com/story/article-name
- Redirects
- Working redirects: this goes here for Molly Henneberg.
- Soft redirects (redirects to /category/): this soft redirects to here. The working URL for this Timeline of the 2003 invasion of Iraq article is here.
- Redirects to 404 pages: this link doesn't work for British debate over veils. The working URL is here.
- Redirects to wrong pages: I found redirects pointing to the wrong article at Adam Levine (press aide), Catherine Herridge and Wood Green ricin plot. They do not follow the foxnews.com/story/article-name format. They instead point to articles at /entertainment/, media/ and /science/. Since these are wrong, I recommend the following URL changes below.
- URL Changes:
- With the above links, the numeric value is changed to the article title. Any punctuation marks are removed from the URL and all letters are lowercase.
- For redirects that do not point to articles using /story/, I request trying to convert them using /story/article-name first. If that doesn't work, then I recommend archive URLs.
~3,200 articles.
Thank you! MrLinkinPark333 (talk) 20:48, 12 September 2024 (UTC)
Enwiki
- Checked 3,248 pages and edited 2,346 pages. Moved 2,601 links to a new URL. Resolved 66 ghost redirects. Resolved 233 soft-404s. Removed 4
{{dead link}}. Added 6{{dead link}}. Switched 900|url-status=deadto live. Switched 10|url-status=liveto dead. Added 240 archive URLs (198 Wayback). Changed 175 citation metadata.
- Analysis: converted about 3,500 to live URLs per the above rules (2,601 + 900). Another 250 or so added archive URLs. -- GreenC 18:07, 30 September 2024 (UTC)
- Not bad at all! How successful were fixing the redirects to wrong pages? MrLinkinPark333 (talk) 18:10, 30 September 2024 (UTC)
- It seems successful. A spot check of Disappearance of Natalee Holloway saw some. -- GreenC 21:25, 30 September 2024 (UTC)
IABot DB
- Checked and updated about 5,700 links that propagate to 300+ wikis.
Done -- GreenC 04:25, 2 October 2024 (UTC)
location.teamname.mlb.com
Each of the 30 MLB teams has a dead subdomain of the form <location>.<teamname>.mlb.com that should be archived, for example losangeles.angels.mlb.com. These now redirect to sites of the form mlb.com/<teamname>, and all content in the subdomains seems to be dead.
I combined the searches into 6 batches of 5 teams each, as combining all teams into one regex expression timed out the search and I didn't want to individually list the results for all 30 teams. I hope it isn't too difficult to process 30 different subdomains?
(Also, for some reason the searches counted a few pages where the text happened to contain <teamname>|mlb.com instead of <teamname>.mlb.com.)
- (a regex "." means match any character thus it matched on "|" or whatever character; to search on a literal dot use "[.]" or "\." to escape the regex meaning of dot) -- GreenC 00:18, 3 October 2024 (UTC)
diamondbacks, braves, orioles, redsox, cubs: 1,305 pages.
whitesox, reds, indians, rockies, tigers: 1,181 pages.
astros, royals, angels, dodgers, marlins: 1,134 pages.
brewers, twins, mets, yankees, athletics: 1,118 pages.
phillies, pirates, padres, giants, mariners: 1,304 pages.
cardinals, rays, devilrays (both are subdomains for the same team), rangers, bluejays, nationals: 1,260 pages. Helpful Raccoon (talk) 05:16, 14 September 2024 (UTC)
- Should be OK to combine into a single project since they use the same root domain, problems like soft-404s will be the same. Thanks for creating the separate searches. I saw one for "m.cubs.mlb.com" which is the mobile link for the Cubs. It is a soft-404, so looks like "*.cubs.mlb.com" need to be checked. -- GreenC 15:54, 14 September 2024 (UTC)
Enwiki
- Checked 5,505 pages and edited 4,080 pages. Moved 4 links to a new URL. Added 4,124
{{dead link}}. Switched 1,160|url-status=liveto dead. Added 5,495 archive URLs (5,431 Wayback). Changed 721 citation metadata.
- Comment: high number of
{{dead link}}-- GreenC 21:27, 3 October 2024 (UTC)- Looks like WaybackMachine performance has been poor creating timeouts resulting in false negatives thus the high number of
{{dead link}}. I am beginning to reprocessing those at a slower pace. -- GreenC 15:35, 5 October 2024 (UTC)
- Looks like WaybackMachine performance has been poor creating timeouts resulting in false negatives thus the high number of
- Comment: high number of
- Round 2: Checked 1,921 pages and edited 1,426 pages. Added 2,388 archive URLs (2,388 Wayback).
- Reprocessed the "Added 4,124
{{dead link}}" from above, due to Wayback Machine timeouts. Converted 2,388{{dead link}}to archive URLs. -- GreenC 17:59, 6 October 2024 (UTC)
- Reprocessed the "Added 4,124
IABot DB
- Checked and updated about 30,000 links which propagate to 300+ wikis
Done -- GreenC 14:14, 8 October 2024 (UTC)
Usurpation: HuskerJ.com
Reporting Wikipedia:Link rot/Usurpations
Site: https://www.huskerj.com/
Linked to / cited from various Nebraska football articles, such as: Chicago Tribune Fans' Poll
PK-WIKI (talk) 17:40, 17 September 2024 (UTC)
Done in WP:JUDI batch #18 -- GreenC 00:40, 25 September 2024 (UTC)
dnd.wizards.com
https://dnd.wizards.com now mostly redirects to https://www.dndbeyond.com; website was used as a primary source for various D&D articles. It looks like links that start with https://dnd.wizards.com/news/, https://dnd.wizards.com/articles/, https://dnd.wizards.com/dndstudioblog, https://dnd.wizards.com/dungeons-and-dragons, etc redirect to the D&D Beyond home page or change log. Some (like https://dnd.wizards.com/products/) redirect to similar pages on D&D Beyond but the D&D Beyond page often contains less information (such as not having the ISBN, author credits or other production info) so I think the whole lot should be marked as dead. Thanks! Sariel Xilo (talk) 22:29, 20 September 2024 (UTC)
159 pages -- GreenC 04:01, 21 September 2024 (UTC)
Enwiki
- Checked 172 pages and edited 150 pages. Added 3
{{dead link}}. Switched 65|url-status=liveto dead. Added 169 archive URLs (159 Wayback). Changed 413 citation metadata.
IABot DB
- Checked and fixed about 500 links which propagate to 300+ wikis
Done -- GreenC 01:37, 7 October 2024 (UTC)
Some Vietnamese newspapers
RFI Vietnamese, VTC News and Zing News changed their domain names:
- vi.rfi.fr and viet.rfi.fr -> rfi.fr/vi
- vtc.vn -> vtcnews.vn
- news.zing.vn and zingnews.vn -> znews.vn
Billboard Vietnam website (billboardvn.vn) has been closed. Cherry Cotton Candy (talk) 09:05, 22 September 2024 (UTC)
vi.rfi.fr
12 pages — Preceding unsigned comment added by GreenC (talk • contribs)
- Tried this to that it doesn't work. -- GreenC 01:41, 7 October 2024 (UTC)
- @GreenC Can you skip the above link and continue with the others? For example, http://vi.rfi.fr/viet-nam/20191111-nhung-nguoi-linh-viet-nam-hy-sinh-vi-nuoc-phap-trong-the-chien-i -> https://www.rfi.fr/vi/viet-nam/20191111-nhung-nguoi-linh-viet-nam-hy-sinh-vi-nuoc-phap-trong-the-chien-i Cherry Cotton Candy (talk) 13:09, 7 October 2024 (UTC)
- Cherry, there are only 12. Could you do this manually? It will be less work than me programming the bot and working through the issues. -- GreenC 15:31, 7 October 2024 (UTC)
vtc.vn
197 pages — Preceding unsigned comment added by GreenC (talk • contribs)
- Checked 198 pages and edited 184 pages. Moved 248 links to a new URL. Added 2
{{dead link}}. Switched 3|url-status=deadto live. Switched 2|url-status=liveto dead. Added 15 archive URLs (11 Wayback). -- GreenC 04:23, 7 October 2024 (UTC)
- Checked and fixed about 2,000 URLs in the IABot DB which will propagate to 300+ wikis. -- GreenC 21:12, 7 October 2024 (UTC)
zingnews.vn
246 pages — Preceding unsigned comment added by GreenC (talk • contribs)
- Checked 246 pages and edited 244 pages. Moved 478 links to a new URL. Removed 2
{{dead link}}. Switched 113|url-status=deadto live. Added 9 archive URLs (4 Wayback). -- GreenC 21:08, 7 October 2024 (UTC)
- Checked and fixed about 2,600 links in the IABot DB which propagate to 300+ wikis -- GreenC 01:42, 8 October 2024 (UTC)
billboardvn.vn and thanhniennews.com
Billboard 130 pages — Preceding unsigned comment added by GreenC (talk • contribs)
Thanhniennews 261 pages. These websites have been closed. Cherry Cotton Candy (talk) 03:59, 23 September 2024 (UTC)
- Checked 389 pages and edited 261 pages. Added 61
{{dead link}}. Switched 92|url-status=liveto dead. Added 178 archive URLs (139 Wayback). -- GreenC 16:25, 8 October 2024 (UTC) - Checked and fixed about 1,000 URLs in the IABot DB which propagate to 300+ wikis -- GreenC 18:18, 8 October 2024 (UTC)
tuoitre.com.vn
41 pages. Some articles can be found manually on tuoitre.vn, for example:
- http://www.tuoitre.com.vn/Tianyon/Index.aspx?ArticleID=159942 -> https://tuoitre.vn/ky-nam-va-tram-huong-159942.htm
- http://bauchon.tuoitre.com.vn/Tianyon/Index.aspx?ArticleID=226165&ChannelID=458 -> https://tuoitre.vn/bao-ton-da-dang-sinh-hoc-vinh-ha-long-226165.htm
- http://www.tuoitre.com.vn/Tianyon/Index.aspx?ArticleID=14965&ChannelID=124 -> https://cuoituan.tuoitre.vn/thap-da-vinh-nghiem-14965.htm
Cherry Cotton Candy (talk) 03:59, 23 September 2024 (UTC)
- Unable to do by bot. -- GreenC 23:58, 7 October 2024 (UTC)
thanhnien.com.vn
124 pages. Some articles can be found manually on thanhnien.vn, for example:
- http://www.thanhnien.com.vn/van-hoa-nghe-thuat/tri-thuc-cung-nghe-nhac-vang-181742.html -> https://thanhnien.vn/tri-thuc-cung-nghe-nhac-vang-185171992.htm
- http://www.thanhnien.com.vn/pages/20120819/tu-bat-pho-khong-nguoi-lai.aspx -> https://thanhnien.vn/tu-bat-pho-khong-nguoi-lai-185406399.htm
- http://www.thanhnien.com.vn/Pages/20110626/Phim-Hollywood-thang-lon-tai-Viet-Nam.aspx -> https://thanhnien.vn/phim-hollywood-thang-lon-tai-viet-nam-185293878.htm
- http://web.thanhnien.com.vn/Khoahoc/2005/9/6/121442.tno -> https://thanhnien.vn/nhung-phat-hien-chan-dong-ve-phong-nha-ke-bang-185148509.htm
- http://www.thanhnien.com.vn/News/Pages/201006/20100204224010.aspx -> https://thanhnien.vn/khanh-hoa-khong-to-chuc-miss-world-2010-185296179.htm
Cherry Cotton Candy (talk) 03:59, 23 September 2024 (UTC)
- Unable by bot. -- GreenC 23:58, 7 October 2024 (UTC)
laodong.com.vn
49 pages. Few articles can be found manually on laodong.vn, for example:
- http://tamlongvang.laodong.com.vn/the-gioi/banh-mi-viet-nam-va-hanh-trinh-chinh-phuc-ca-the-gioi-591142.bld -> https://laodong.vn/an/banh-mi-viet-nam-va-hanh-trinh-chinh-phuc-ca-the-gioi-515042.ldo
- http://laodong.com.vn/giai-tri/lan-khue-thu-suc-lam-mc-365442.bld -> https://laodong.vn/archived/lan-khue-thu-suc-lam-mc-693704.ldo
Cherry Cotton Candy (talk) 03:59, 23 September 2024 (UTC)
- Unable by bot. -- GreenC 23:58, 7 October 2024 (UTC)
Done -- GreenC 18:19, 8 October 2024 (UTC)
aviation-safety.net
These (currently) 299 results ought to have "/operator/airline.php?var=" replaced by "/operators/". Updating the redirected domain "aviation-safety.net" to "asn.flightsafety.org" could be done along the way as well. 1234qwer1234qwer4 16:02, 24 September 2024 (UTC)
- User:1234qwer1234qwer4, given http://aviation-safety.net/database/operator/airline.php?var=6345 can you tell me the new URL? -- GreenC 16:07, 24 September 2024 (UTC)
- http://aviation-safety.net/database/operators/6345 works, though it is a redirect to https://asn.flightsafety.org/database/operators/6345. 1234qwer1234qwer4 16:13, 24 September 2024 (UTC)
Enwiki
- Checked 298 pages and edited 298 pages. Moved 1,073 links to a new URL. Resolved 8 ghost redirects. Switched 7
|url-status=deadto live. Switched 2|url-status=liveto dead. Added 22 archive URLs (21 Wayback).
IABot DB
- Checked and fixed about 800 links which propagate across 300+ wikis.
Done -- GreenC 22:52, 8 October 2024 (UTC)
planespotters.net
260 pages that should have "planespotters.net/Airline/" changed to "planespotters.net/airline/". 1234qwer1234qwer4 17:16, 24 September 2024 (UTC)
- Checked 241 pages and edited 231 pages. Moved 251 links to a new URL. Removed 1
{{dead link}}. Added 1{{dead link}}. Switched 99|url-status=deadto live. Added 22 archive URLs (13 Wayback).
Done -- GreenC 23:13, 8 October 2024 (UTC)
articles.newspaper.com
Newspapers that follow the same process as Tribune Publishing. Only migrate if new links are not behind paywall, otherwise archive. -- GreenC 18:01, 24 September 2024 (UTC)
articles.baltimoresun.com
- Enwiki
- Checked 4,150 pages and edited 3,977 pages. Converted 1 templates. Moved 4,833 links to a new URL. Resolved 51 ghost redirects. Resolved 11 soft-404s. Removed 3
{{dead link}}. Added 64{{dead link}}. Switched 791|url-status=deadto live. Switched 39|url-status=liveto dead. Added 688 archive URLs (494 Wayback). Changed 50 citation metadata.
- Checked 4,150 pages and edited 3,977 pages. Converted 1 templates. Moved 4,833 links to a new URL. Resolved 51 ghost redirects. Resolved 11 soft-404s. Removed 3
- IABot DB
- Checked and updated about 8,000 URLs which propagate to 300+ wikis
- Done -- GreenC 03:35, 2 November 2024 (UTC)
articles.timesofindia.indiatimes.com
- Enwiki
- Pass 1: Checked 9,455 pages and edited 1,877 pages. Moved 1,745 links to a new URL. Resolved 1,663 ghost redirects. Resolved 10 soft-404s. Removed 10
{{dead link}}. Added 133{{dead link}}. Switched 1,621|url-status=deadto live. Switched 80|url-status=liveto dead. Added 92 archive URLs (92 Wayback). Changed 334 citation metadata.
- Pass 1: Checked 9,455 pages and edited 1,877 pages. Moved 1,745 links to a new URL. Resolved 1,663 ghost redirects. Resolved 10 soft-404s. Removed 10
- Pass 2 after improvements to Ghost redirect code: Checked 9,455 pages and edited 3,488 pages. Moved 5,186 links to a new URL. Discovered 5,126 ghost redirects. Resolved 229 soft-404s. Removed 7
{{dead link}}. Switched 5,061|url-status=deadto live. Switched 1|url-status=liveto dead. Added 16 archive URLs (0 Wayback). Changed 1 citation metadata. -- GreenC 18:24, 15 October 2024 (UTC)
- Pass 2 after improvements to Ghost redirect code: Checked 9,455 pages and edited 3,488 pages. Moved 5,186 links to a new URL. Discovered 5,126 ghost redirects. Resolved 229 soft-404s. Removed 7
- Pass 3: Checked 9,455 pages and edited 1,359 pages. Moved 1,839 links to a new URL. Discovered 1,780 ghost redirects. Removed 2
{{dead link}}. Switched 1,762|url-status=deadto live. Added 4 archive URLs (0 Wayback).
- Pass 3: Checked 9,455 pages and edited 1,359 pages. Moved 1,839 links to a new URL. Discovered 1,780 ghost redirects. Removed 2
- Pass 4: Checked 9,455 pages and edited 276 pages. Moved 378 links to a new URL. Discovered 331 ghost redirects. Removed 1
{{dead link}}. Switched 323|url-status=deadto live. Added 7 archive URLs (0 Wayback).
- Pass 4: Checked 9,455 pages and edited 276 pages. Moved 378 links to a new URL. Discovered 331 ghost redirects. Removed 1
- IABot DB
- Checked and updated
- Done -- GreenC 18:31, 4 November 2024 (UTC)
articles.economictimes.indiatimes.com
- Enwiki
- Checked 2,654 pages and edited 2,422 pages. Moved 86 links to a new URL. Discovered 3 ghost redirects. Added 115
{{dead link}}. Switched 587|url-status=liveto dead. Added 2,505 archive URLs (2,237 Wayback). Changed 23 citation metadata.
- Checked 2,654 pages and edited 2,422 pages. Moved 86 links to a new URL. Discovered 3 ghost redirects. Added 115
- IABot DB
- Checked and updated
- Done -- GreenC 18:31, 4 November 2024 (UTC)
articles.philly.com
- Enwiki
- Checked 4,702 pages and edited 4,055 pages. Resolved 2,032 soft-404s. Added 75
{{dead link}}. Switched 550|url-status=liveto dead. Added 5,215 archive URLs (4,721 Wayback). Changed 160 citation metadata.
- Analysis: the URLs were unable to convert via the
|title=method, like elsewhere with the other articles.* domains. It has ghost redirects, but they are all soft-404s pointing to the home page. Last option was archive URLs, which it was mostly able, except for 75{{dead link}}.
- Analysis: the URLs were unable to convert via the
- Checked 4,702 pages and edited 4,055 pages. Resolved 2,032 soft-404s. Added 75
- IABot DB
- Checked and updated
- Done -- GreenC 18:31, 4 November 2024 (UTC)
articles.sfgate.com
1,300 pages GreenC 18:01, 24 September 2024 (UTC)
- Enwiki
- Pass 1: Checked 1,365 pages and edited 1,328 pages. Moved 1,236 links to a new URL. Discovered 1,236 ghost redirects. Resolved 4 soft-404s. Removed 76
{{dead link}}. Added 26{{dead link}}. Switched 102|url-status=deadto live. Switched 23|url-status=liveto dead. Added 291 archive URLs (122 Wayback). Changed 7 citation metadata. - Pass 2: Checked 1,365 pages and edited 241 pages. Moved 266 links to a new URL. Discovered 266 ghost redirects. Removed 25
{{dead link}}. Switched 234|url-status=deadto live. Added 1 archive URLs (1 Wayback).
- Pass 1: Checked 1,365 pages and edited 1,328 pages. Moved 1,236 links to a new URL. Discovered 1,236 ghost redirects. Resolved 4 soft-404s. Removed 76
- IABot DB
- Checked and updated
- Done -- GreenC 18:31, 4 November 2024 (UTC)
articles.washingtonpost.com
- Enwiki
- Pass 1: Checked 774 pages and edited 444 pages. Moved 426 links to a new URL. Discovered 426 ghost redirects. Removed 13
{{dead link}}. Added 28{{dead link}}. Switched 396|url-status=deadto live. Switched 1|url-status=liveto dead. Added 12 archive URLs (4 Wayback). Changed 8 citation metadata. - Pass 2: Checked 774 pages and edited 141 pages. Moved 126 links to a new URL. Discovered 126 ghost redirects. Switched 124
|url-status=deadto live.
- Pass 1: Checked 774 pages and edited 444 pages. Moved 426 links to a new URL. Discovered 426 ghost redirects. Removed 13
- IABot DB
- Checked and updated
- Done -- GreenC 18:31, 4 November 2024 (UTC)
articles.boston.com
- Enwiki
- Pass 1: Checked 622 pages and edited 263 pages. Moved 103 links to a new URL. Discovered 103 ghost redirects. Removed 1
{{dead link}}. Added 63{{dead link}}. Switched 76|url-status=deadto live. Switched 6|url-status=liveto dead. Added 101 archive URLs (50 Wayback). Changed 7 citation metadata. - Pass 2: Checked 622 pages and edited 26 pages. Moved 26 links to a new URL. Discovered 26 ghost redirects. Switched 24
|url-status=deadto live.
- Pass 1: Checked 622 pages and edited 263 pages. Moved 103 links to a new URL. Discovered 103 ghost redirects. Removed 1
- IABot DB
- Checked and updated
- Done -- GreenC 18:31, 4 November 2024 (UTC)
articles.herald-mail.com
- Enwiki
- Checked 123 pages and edited 75 pages. Moved 4 links to a new URL. Added 4
{{dead link}}. Switched 2|url-status=liveto dead. Added 88 archive URLs (78 Wayback).
- Checked 123 pages and edited 75 pages. Moved 4 links to a new URL. Added 4
- IABot DB
- Checked and updated
- Done -- GreenC 18:31, 4 November 2024 (UTC)
articles.businessinsider.com
"has a paywall"
- Enwiki
- Checked 133 pages and edited 23 pages. Added 1
{{dead link}}. Switched 1|url-status=liveto dead. Added 18 archive URLs (9 Wayback). Changed 1 citation metadata.
- Checked 133 pages and edited 23 pages. Added 1
- IABot DB
- Checked and updated
- Done -- GreenC 18:31, 4 November 2024 (UTC)
articles.dailypilot.com
"redirect to latimes and don't appear to have easy conversion rules"
- Enwiki
- Checked 109 pages and edited 88 pages. Added 2
{{dead link}}. Switched 10|url-status=liveto dead. Added 113 archive URLs (110 Wayback).
- Checked 109 pages and edited 88 pages. Added 2
- IABot DB
- Checked and updated
- Done -- GreenC 18:31, 4 November 2024 (UTC)
singapore-elections.com
website is dead. hostile takeover by the usual.. casino suspects. – robertsky (talk) 02:55, 26 September 2024 (UTC)
Done in WP:JUDI batch #19 -- GreenC 17:56, 5 November 2024 (UTC)
ittiofauna.org
Has been WP:JUDI usurped by a Thai site. Redirects to Gbo5000 - Mainkan Slot Gacor Server Thailand Resmi (dacres.org)
It used to contain photos of European fish, and there are ~18 occurrences in the articlespace. Big Blue Cray(fish) Twins (talk) 12:51, 27 September 2024 (UTC)
Done in WP:JUDI batch #19 -- GreenC 17:56, 5 November 2024 (UTC)
usemod.com
Domain is usurped, new domain is usemod.org. Paths should work the same. * Pppery * it has begun... 04:44, 30 September 2024 (UTC)
3 pages. I edited them manually.
Done -- GreenC 18:03, 5 November 2024 (UTC)
ctv.ca
Hello. Old CTV links don't work anymore. I did not find any that were now at ctvnews.ca, Therefore, I request archives for these links only. ~1500 articles. Some of these already have archives added in the article. Thanks! MrLinkinPark333 (talk) 22:47, 30 September 2024 (UTC)
- I did this domain in 2021. Developed code to move links to ctvnews.ca .. example diff: Special:Diff/1029587503/1033596569 .. converted about 1,000 links. But that code won't work anymore as the redirect information no longer exists. Currently there are 1,847 pages. I'll try to find ghost redirects otherwise convert to archive. -- GreenC 23:45, 5 November 2024 (UTC)
- That's too bad that the conversion doesn't work anymore. Hopefully some more can be changed over if possible. MrLinkinPark333 (talk) 23:50, 5 November 2024 (UTC)
- Enwiki
- Checked 1,854 pages and edited 459 pages. Moved 2 links to a new URL. Resolved 6 soft-404s. Switched 4
|url-status=liveto dead. Added 133 archive URLs (116 Wayback). Changed 184 citation metadata.- Analysis - It found only 2 conversions. The 133 archive URLs might be links that were missed in 2021 due to improvements in code, new archives at the service provider, or new links (re)added to Wikipedia since 2021. The citation metadata is a new feature not available in 2021. Overall, it looks about expected. -- GreenC 16:42, 6 November 2024 (UTC)
- Checked 1,854 pages and edited 459 pages. Moved 2 links to a new URL. Resolved 6 soft-404s. Switched 4
- IABot DB
- Previously done and for any stragglers I changed the status to permadead in the DB
Done -- GreenC 16:42, 6 November 2024 (UTC)
citynews.ca
Hello again. citynews.ca links are mostly redirecting to toronto.citynews.ca:
- Links with dates: this is now here for The Christmas Shoes (song).
- Links without dates: Other article links need to be converted to toronto.citynews.ca/year/month/day/name-of-article/ - For example this is now here for First Canadian Place. Unfortunately, the old URL does not have the date already listed, so it either has to be extracted from the citation or archived copy. Any punctuation marks are removed
~290 links. If any of these new links do not work, it is possible that it's under a different subdomain like calgary.citynews.ca. As Toronto is the main domain, it might be easier to test if they convert to toronto.citynews.ca, then archive the ones that don't work. Please let me know if any of these don't convert to new links. Thanks! MrLinkinPark333 (talk) 23:28, 30 September 2024 (UTC)
- MrLinkinPark333: Let me know what you want to do with the below 37. If not too complicated. At some point it's easier to fix small numbers by hand. Overall it found most of them successfully Special:Diff/1255156660/1255809021. It scraped the title from the citation
|title=, and scraped the date from the archive URL page content; reformatted and assembled into a new URL. -- GreenC 20:02, 6 November 2024 (UTC)- Were any of them converted to calgary? I used that as an example as there are 9 subdomains. However, I think that the rest of them would be at toronto and just need manually converting. Let me know if that's the case, and I'll swap the rest manually. MrLinkinPark333 (talk) 20:47, 6 November 2024 (UTC)
- Converted only 1 calgary Special:Diff/1255149041/1255811790 -- GreenC 21:08, 6 November 2024 (UTC)
- Were any of them converted to calgary? I used that as an example as there are 9 subdomains. However, I think that the rest of them would be at toronto and just need manually converting. Let me know if that's the case, and I'll swap the rest manually. MrLinkinPark333 (talk) 20:47, 6 November 2024 (UTC)
- MrLinkinPark333: Let me know what you want to do with the below 37. If not too complicated. At some point it's easier to fix small numbers by hand. Overall it found most of them successfully Special:Diff/1255156660/1255809021. It scraped the title from the citation
Enwiki
- Pass 1: toronto and calgary: Checked 298 pages and edited 266 pages. Moved 291 links to a new URL. Removed 1
{{dead link}}. Added 1{{dead link}}. Switched 39|url-status=deadto live. Added 10 archive URLs (5 Wayback). Changed 4 citation metadata.
Articles that still have www.citynews.ca links after trying conversion to toronto or calgary.citynews.ca
IABOt DB
- Checked and updated.
Done -- GreenC 01:56, 8 November 2024 (UTC)
deseretnews.com
Almost all links here are soft redirects to articles at deseret.com, but conversion seems to be intractable, so the links should be archived. The converted links are of the form www.deseret.com/year/month/day/<id>/title-of-article, where the <id> seems to be unrelated to anything in the old link. Example: link [22] in 2012 United States presidential election is a soft redirect to [23].
5,446 pages. Helpful Raccoon (talk) 02:48, 6 October 2024 (UTC)
Enwiki
- Checked 5,456 pages and edited 4,934 pages. Moved 741 links to a new URL. Of which 736 are ghost redirects. Resolved 19 soft-404s. Removed 1
{{dead link}}. Added 240{{dead link}}. Switched 60|url-status=deadto live. Switched 628|url-status=liveto dead. Added 6,336 archive URLs (5,748 Wayback). Changed 342 citation metadata.
IABot
- Checked and updated
Done -- GreenC 15:47, 8 November 2024 (UTC)
foxnews.com/section/year/
Fox News articles of the form foxnews.com/<section>/yyyy/mm/dd/.... are soft redirects to articles of the form foxnews.com/<section>/title-of-article. Example: [24] in "Weird Al" Yankovic is a soft redirect to [25] (note that the text at the end of the first URL differs from that of the second, with "adapting" apparently misspelled in the first). Conversion is usually tractable so long as the article title is known, as it is similar to the Chicago Tribune conversion.
7,259 pages. Helpful Raccoon (talk) 03:14, 6 October 2024 (UTC)
- Looks like two types of conversions: a simple URL transform by removing the date; and the harder "Chicago method", of extracting the title from the citation. I guess the best way is try to simple method first and if not then the Chicago method; if those do not work then check for ghost redirects; and finally add an archive. -- GreenC 15:59, 8 November 2024 (UTC)
- It's working, but took a while to code as this is the first time I've attempted sequencing all the methods at once. The "Chicago" method is still pretty custom, I need to integrate it as part of the boilerplate code as a standard feature. Also with all these methods it's slow, 7,000 pages will take a while. -- GreenC 19:58, 8 November 2024 (UTC)
- I added two new concepts to the glossary: ruled soft-redirect, and inferred soft-redirect. In this case, the removal of the date from the URL is a 'ruled soft-redirect' ie. a hard-coded rule to transform the URL. The parsing of the title is an 'inferred soft-redirect' because it is inferring (guessing) what the new URL might be, and could generate multiple guesses into an 'inference table', from which the bot checks each guess, until it finds a match. The inferred soft-redirect code is now incorporated as a feature that can be enabled/disabled for each project. -- GreenC 06:26, 9 November 2024 (UTC)
- Helpful Raccoon, thanks for finding and reporting Fox News, it was helpful on a couple levels. Fixing the links, improving the bot's general code for future domains, and helping to distinguish (or at least name) the concepts of 'ruled soft-redirects' and 'inferred soft-redirects'. -- GreenC 15:14, 10 November 2024 (UTC)
- I added two new concepts to the glossary: ruled soft-redirect, and inferred soft-redirect. In this case, the removal of the date from the URL is a 'ruled soft-redirect' ie. a hard-coded rule to transform the URL. The parsing of the title is an 'inferred soft-redirect' because it is inferring (guessing) what the new URL might be, and could generate multiple guesses into an 'inference table', from which the bot checks each guess, until it finds a match. The inferred soft-redirect code is now incorporated as a feature that can be enabled/disabled for each project. -- GreenC 06:26, 9 November 2024 (UTC)
- It's working, but took a while to code as this is the first time I've attempted sequencing all the methods at once. The "Chicago" method is still pretty custom, I need to integrate it as part of the boilerplate code as a standard feature. Also with all these methods it's slow, 7,000 pages will take a while. -- GreenC 19:58, 8 November 2024 (UTC)
Enwiki
- Checked 7,259 pages and edited 6,849 pages. Moved 7,986 links to a new URL. Of which 477 were ghost redirects; 4,276 were inferred soft-redirects; 2,894 were ruled soft-redirects; 539 were regular redirects. Added 565 archive URLs (462 Wayback).
IABot DB
- Checked and updated about 15,000 URLs which propagate to 300+ wikis
Done -- GreenC 15:14, 10 November 2024 (UTC)
cnbc.com/id/number/title
Articles of the form cnbc.com/id/<eight digit id>/<article title> can be converted to live articles or redirects by simply removing everything after the 8-digit id. Example: https://www.cnbc.com/id/37207942/Could_Italy_Be_Better_Off_than_its_Peers in Italy can be converted to https://www.cnbc.com/id/37207942, which redirects to the live article https://www.cnbc.com/2010/05/18/could-italy-be-better-off-than-its-peers.html.
A different example: https://www.cnbc.com/id/47387334/Jim_Breyer_via_Accel_Partners from Facebook can be converted to https://www.cnbc.com/id/47387334, which is a live article.
1,644 pages. Helpful Raccoon (talk) 08:23, 6 October 2024 (UTC)
- OK. Some redirect some do not. I'll test them all and migrate the ones that redirect. It increased the search size, since it's also including anything with only an ID number. -- GreenC 16:24, 10 November 2024 (UTC)
Enwiki
- Checked 1,654 pages and edited 1,491 pages. Moved 1,492 links to a new URL: 1,389 ruled soft-redirects, 103 ghost soft-redirects. Resolved 22 soft-404s. Removed 1
{{dead link}}. Added 140{{dead link}}. Switched 107|url-status=deadto live. Switched 10|url-status=liveto dead. Added 142 archive URLs (114 Wayback). Changed 305 citation metadata.
Done -- GreenC 01:18, 11 November 2024 (UTC)
newamericamedia.org
217 pages. New American Media has ceased operations. Links to its website no longer work and its domain name may have been taken over. Cherry Cotton Candy (talk) 03:11, 8 October 2024 (UTC)
- Hijacked. I added it to WP:JUDI. thanks!
ω Awaiting next JUDI batch. -- GreenC 01:27, 11 November 2024 (UTC)
en.rsf.org
567 pages. This website always returns the error code 521. Cherry Cotton Candy (talk) 03:25, 8 October 2024 (UTC)
Enwiki
- Checked 565 pages and edited 257 pages. Added 3
{{dead link}}. Switched 67|url-status=liveto dead. Added 286 archive URLs (246 Wayback). Changed 2 citation metadata.
IABot DB
- Checked and done a few thousand.
Done -- GreenC 15:51, 11 November 2024 (UTC)
variety.com
Links with parameters do not work. If parameters are removed, some links will become redirect links.
- https://www.variety.com/article/VR1118016497?refCatId=16 does not work.
- https://www.variety.com/article/VR1118016497 redirects to https://variety.com/2010/film/markets-festivals/willie-nelson-launches-luck-films-1118016497/
- https://variety.com/article/VR102012.html?categoryid=4&cs=1&query=garth+brooks does not work.
- https://variety.com/article/VR102012 redirects to https://variety.com/1992/voices/columns/that-was-the-year-that-was-a-wrap-song-for-92-102012/
Cherry Cotton Candy (talk) 04:28, 8 October 2024 (UTC)
Enwiki
- Checked 2,852 pages and edited 2,681 pages. Moved 4,468 links to a new URL: 4,468 ruled soft-redirects. Removed 24
{{dead link}}. Added 6{{dead link}}. Switched 554|url-status=deadto live. Switched 18|url-status=liveto dead. Added 106 archive URLs (53 Wayback). Changed 178 citation metadata.
IABot DB
- Checked and updated about 14,000 links which propagate to 300+ wikis
Done -- GreenC 15:59, 12 November 2024 (UTC)
kotaku.com.au
1357 pages for https://www.kotaku.com.au - Kotaku Australia is now redirecting to Kotaku's front page (see update on Aftermath). Sariel Xilo (talk) 23:55, 15 October 2024 (UTC)
Enwiki
- Checked 1,370 pages and edited 1,303 pages. Added 8
{{dead link}}. Switched 825|url-status=liveto dead. Added 772 archive URLs (749 Wayback). Changed 89 citation metadata.
IABot DB
- Checked and updated about 2,000 links which propagate to 300+ wikis
Done -- GreenC 23:15, 12 November 2024 (UTC)
community.seattletimes.nwsource.com
All of the "http://community.seattletimes.nwsource.com" links seem to be dead, but can be substituted with "https://archive.seattletimes.com" as seen in Special:Diff/1253654883
There are 2,943 articles that match this description: per this search result.
I tried this with several links and it seemed to work fine. I'm not sure how many failed the transfer, but testing a bunch and it being fine seems to me like a lot of them still exist.
Take for instance, the one provided in the Gulf War page: http://community.seattletimes.nwsource.com/archive/?date=19910912&slug=1305069
An archive does exist, and it shows what is shown with the url replacement: Archived old link vs Live updated link Chewsterchew (talk) 04:59, 27 October 2024 (UTC)
Enwiki
- Checked 2,951 pages and edited 2,905 pages. Moved 4,195 links to a new URL: 3,954 ruled soft-redirects, Removed 5
{{dead link}}. Switched 287|url-status=deadto live. Added 33 archive URLs (20 Wayback). Changed 255 citation metadata.
IABot DB
- Checked and updated about 1,000 links
Done -- GreenC 04:04, 13 November 2024 (UTC)
disneyparks.disney.go.com/blog/
"disneyparks.disney.go.com/blog/" redirects to https://disneyparksblog.com/, with none of the articles/post still active/archived. I've tried to {{dead link}} many of them and have submitted for InternetArchiveBot to run on many of the pages, but I'm sure I missed a bunch of them as well. Elisfkc (talk) 02:55, 28 October 2024 (UTC)
Enwiki
- Checked 461 pages and edited 390 pages. Removed 1
{{dead link}}. Added 7{{dead link}}. Switched 156|url-status=liveto dead. Added 544 archive URLs (520 Wayback). Changed 3 citation metadata.
IABot DB
- Checked and updated about 900 URLs that will propagate to 300+ wikis
Done -- GreenC 15:18, 13 November 2024 (UTC)
avclub.com/articles
Seems like a lot of their music reviews have dead links. How can we fix this? Cahlin29 (talk) 03:58, 30 October 2024 (UTC)
- Is there an example? -- GreenC 04:30, 30 October 2024 (UTC)
- The link on Drake's Take Care is dead: https://www.avclub.com/articles/drake-take-care,65046
- Same with Mac & Devin Go to High School (soundtrack): https://www.avclub.com/articles/snoop-dogg-and-wiz-khalifa-mac-and-devin-go-to-hig,66410
- Also with Curtis (50 Cent album): https://www.avclub.com/articles/50-cent-curtis,7557
- I'm presuming a pattern. Cahlin29 (talk) 17:22, 30 October 2024 (UTC)
- The Drake link was moved here. The number "1798170489" is the key. I was able to find it in a ghost redirect as seen here (the old URL redirects to the new URL). It will be a while, I need to get through everything else above first. Looks like about 4,600 pages. -- GreenC 17:55, 30 October 2024 (UTC)
- No worries, take your time, I assume the Internet Archive outage delayed things. Cahlin29 (talk) 20:45, 30 October 2024 (UTC)
- The Drake link was moved here. The number "1798170489" is the key. I was able to find it in a ghost redirect as seen here (the old URL redirects to the new URL). It will be a while, I need to get through everything else above first. Looks like about 4,600 pages. -- GreenC 17:55, 30 October 2024 (UTC)
Enwiki
- First pass: Checked 4,601 pages and edited 2,924 pages. Moved 3,133 links to a new URL: 3,133 ghost soft-redirects. Switched 120
|url-status=deadto live. Added 73 archive URLs (26 Wayback). Changed 770 citation metadata. - Second pass: Checked 2,607 pages and edited 1,751 pages. Moved 3,493 links to a new URL: 468 inferred CDX soft-redirects, 3,025 ghost soft-redirects, Added 9
{{dead link}}. Switched 32|url-status=deadto live. Switched 115|url-status=liveto dead. Added 1,199 archive URLs (1,067 Wayback). Changed 213 citation metadata.
- Analysis: created a new method for discovery: inferred CDX soft-redirects. Converted domain names *.xvclub.com to www.avclub.com. Improved ghost redirect detection
IABot DB
- Updated about 11,000 links that propagate to 300+ wikis
Done - GreenC 05:01, 15 November 2024 (UTC)
empoweringindia.org
655 pages. This domain was sold to a gambling website, and Citation bot changed the titles of these links. Cherry Cotton Candy (talk) 04:11, 3 November 2024 (UTC)
Done in WP:JUDI batch #19 -- GreenC 17:56, 5 November 2024 (UTC)
michmarkers.com
262 pages. It has been usurped by a gambling website. Cherry Cotton Candy (talk) 09:14, 3 November 2024 (UTC)
Done in WP:JUDI batch #19 -- GreenC 17:56, 5 November 2024 (UTC)
ouramericanrevolution.org
Colonial Williamsburg site. 10 pages -- GreenC 19:36, 6 November 2024 (UTC)
- Done - via IABot job. -- GreenC 05:07, 15 November 2024 (UTC)
southdreamz.com
Website has been usurped. Doesn't look like JUDI but it redirects to a completely different website such as the link at Naan Mahaan Alla (2010 film). 73 articles. MrLinkinPark333 (talk) 20:50, 7 November 2024 (UTC)
ω Awaiting next WP:JUDI batch. -- GreenC 05:23, 12 November 2024 (UTC)
screenindia.com
This website soft redirects to indianexpress.com but has no equivalent text. Therefore, this needs archives only. 810 articles. Some of them already have archives added, such as at Vakkalathu Narayanankutty.Thanks! MrLinkinPark333 (talk) 02:24, 8 November 2024 (UTC)
- Technically soft 404 (vs. soft redirect). Corollary concepts. Soft 404 redirects when it shouldn't. Soft redirect doesn't redirect when should. -- GreenC 05:19, 15 November 2024 (UTC)
Enwiki
- Checked 823 pages and edited 340 pages. Added 184
{{dead link}}. Switched 23|url-status=liveto dead. Added 136 archive URLs (104 Wayback). Changed 88 citation metadata.
IABot DB
- Checked and fixed about 400 links which propagate to 300+ wikis
Done -- GreenC 16:33, 15 November 2024 (UTC)
time.com
Time.com has moved their links to new URLs. Unfortunately, they are not easy to convert. For example, this is now here for Paul McCartney.. Therefore, I request archives URLs instead ~20k articles. Some of them already have archives added. Thanks! MrLinkinPark333 (talk) 15:53, 9 November 2024 (UTC)
- I processed time.com in July 2021. It was large, took three days to process. Added 25,000 archive URLs. You can read my strategy in the link. Do you still see a lot of broken links without archive URLs? -- GreenC 01:07, 11 November 2024 (UTC)
- Of the first 500 in the above link, 194 don't show archives. If you could filter out the ones without archive URLs for time, it'll help a lot. MrLinkinPark333 (talk) 01:11, 11 November 2024 (UTC)
- How are you checking for archives? 194 is about 40%. I just manually checked 50 pages, every one has an archive (need to open the page and search on the link, the search result page doesn't provide enough information to determine). Except 3 cases that have a live link. Of those 50, in no cases would the bot add an archive URL. I could do this, but it will take a while to process, and I'm not sure how much it will accomplish. BTW the Paul McCartney example link no longer exists in the article, but it does exist in two others. Both have archives. -- GreenC 19:36, 11 November 2024 (UTC)
- I only checked the results page and not manually checked each individual article. Is it possible to adjust the search result link above to calculate how many articles don't have archives first for time? Then, we could decide what to do next. MrLinkinPark333 (talk) 19:43, 11 November 2024 (UTC)
- There is no easy way for this search. But recall Wikipedia:Link_rot/URL_change_requests#ctv.ca, which was also previously done in 2021, and it found 133 more archives. Maybe it's worth trying again. I'll need to build a list of target articles by searching a dump file, since the online search tops out at 10,000 results. -- GreenC 05:06, 12 November 2024 (UTC)
- If you believe this is easier, feel free to check all of them. Since this request is big, I don't mind if it gets done later after the smaller requests are done. MrLinkinPark333 (talk) 02:16, 13 November 2024 (UTC)
- Extracting all the page names that contain time.com requires searching a dump file which can take 6-8 hours to complete. This is required when the number of results is > 10,000 because Cirrus search (eg. "insource:..") won't return more than 10k results, due to resource constraints on their search server. Cirrus can return how many results there are > 10k, but won't display the actual results beyond 10k. I'll need to do the same with deadline.com below which has 40k results. -- GreenC 19:46, 15 November 2024 (UTC)
- If you believe this is easier, feel free to check all of them. Since this request is big, I don't mind if it gets done later after the smaller requests are done. MrLinkinPark333 (talk) 02:16, 13 November 2024 (UTC)
- There is no easy way for this search. But recall Wikipedia:Link_rot/URL_change_requests#ctv.ca, which was also previously done in 2021, and it found 133 more archives. Maybe it's worth trying again. I'll need to build a list of target articles by searching a dump file, since the online search tops out at 10,000 results. -- GreenC 05:06, 12 November 2024 (UTC)
- I only checked the results page and not manually checked each individual article. Is it possible to adjust the search result link above to calculate how many articles don't have archives first for time? Then, we could decide what to do next. MrLinkinPark333 (talk) 19:43, 11 November 2024 (UTC)
- How are you checking for archives? 194 is about 40%. I just manually checked 50 pages, every one has an archive (need to open the page and search on the link, the search result page doesn't provide enough information to determine). Except 3 cases that have a live link. Of those 50, in no cases would the bot add an archive URL. I could do this, but it will take a while to process, and I'm not sure how much it will accomplish. BTW the Paul McCartney example link no longer exists in the article, but it does exist in two others. Both have archives. -- GreenC 19:36, 11 November 2024 (UTC)
- Of the first 500 in the above link, 194 don't show archives. If you could filter out the ones without archive URLs for time, it'll help a lot. MrLinkinPark333 (talk) 01:11, 11 November 2024 (UTC)
Enwiki
- Checked 44,901 pages and edited 13,920 pages. Moved 14,455 links to a new URL: 14,074 ruled soft-redirects, 381 ghost soft-redirects, Resolved 7,124 soft-404s. Removed 9
{{dead link}}. Switched 660|url-status=deadto live. Added 740 archive URLs (446 Wayback). Changed 2,281 citation metadata.
- Analysis: almost all 'ruled soft redirects' are http -> https. Since 'ghost redirects' were not available in 2021, they were discovered this time. Most of the archive URLs were non-Time.com domains that had a
{{dead link}}tag and repaired incidentally. It was able to convert many|work=time.comto|work=Time, because this feature did not exist in 2021.
- Analysis: almost all 'ruled soft redirects' are http -> https. Since 'ghost redirects' were not available in 2021, they were discovered this time. Most of the archive URLs were non-Time.com domains that had a
Done -- GreenC 02:19, 17 November 2024 (UTC)
- Nice to see many fixes! MrLinkinPark333 (talk) 03:37, 17 November 2024 (UTC)
deadline.com
Deadline.com redirects to new URLs with numeric IDs at the end. Any punctuation marks are removed like at this link to go here for Robert Pattinson. Any links that already have an numeric ID at the end can be skipped. ~1300 articles. Thank you! MrLinkinPark333 (talk) 16:06, 9 November 2024 (UTC)
- There are over 40,000 pages with deadline.com .. limit to www.deadline.com there are 4,780. This is what I am checking on "Pass 1". -- GreenC 17:16, 15 November 2024 (UTC)
Enwiki
- Pass1: Checked 4,784 pages and edited 4,364 pages. Moved 6,575 links to a new URL: 6,575 ruled soft-redirects, Added 24
{{dead link}}. Switched 98|url-status=deadto live. Switched 143|url-status=liveto dead. Added 442 archive URLs (401 Wayback). Changed 1,295 citation metadata. - Pass 2: Checked 39,245 pages and edited 5,808 pages. Moved 2,278 links to a new URL: 2,278 ruled soft-redirects, Added 95
{{dead link}}. Switched 32|url-status=deadto live. Switched 1,119|url-status=liveto dead. Added 2,126 archive URLs (2,018 Wayback). Changed 2,399 citation metadata.
Done -- GreenC 00:22, 18 November 2024 (UTC)
passport.weibo.com
Weibo is a Chinese social media platform with a lot of official information disseminated through the official accounts. Some editors tend to use the visitor landing url with prefix when citing a specific post. So a url cited on Death of Li Keqiang goes like:
Am hoping to clean all the citations, just to take what goes after the 'url=' with '%3A → :' and '%2F → /' so the url becomes https://weibo.com/1938487875/NpL26wys2. NoCringe (talk) 02:24, 11 November 2024 (UTC)
- Hi NoCringe: 134 pages. The links are not dead, but I can still process them for link normalization. And if it finds any are dead it will add an archive. -- GreenC 05:19, 12 November 2024 (UTC)
- Thank you! Please process them. It will make archiving easier since the IABot gets stuck on some of these landing pages. NoCringe (talk) 06:57, 12 November 2024 (UTC)
- NoCringe: This is not working. For example in Liu Duan Duan there is this which translates to that. Which is a soft-404 which I don't know how to detect due to foreign script. Furthermore, if you follow the redirect headers, it actually should go to here, which doesn't even work. There are some legitimate working pages, also many not. Without a way to detect the non-working pages, I can't remove existing URLs. -- GreenC 02:47, 18 November 2024 (UTC)
- Ugh I didn't realise there are variations. I will try to clean them up manually when time permits. Thanks for processing by the way. NoCringe (talk) 08:57, 18 November 2024 (UTC)
- User:NoCringe, ok good luck. I wrote the code to extract and decode the sub-URL, if you need me to run it and log the results I can post a three column table: <wikipage> <origiurl> <extracted url> -- GreenC 15:02, 19 November 2024 (UTC)
- Ugh I didn't realise there are variations. I will try to clean them up manually when time permits. Thanks for processing by the way. NoCringe (talk) 08:57, 18 November 2024 (UTC)
paleobiodb.org
Their former URLs paleodb.org and fossilworks.org have been taken over by The Ecological Register; a seemingly well-meaning site. The old URLs such as:
http://paleodb.org/cgi-bin/bridge.pl?a=checkTaxonInfo&taxon_no=34738
http://www.fossilworks.org/cgi-bin/bridge.pl?a=taxonInfo&taxon_no=64541
have now become:
https://paleobiodb.org/classic/checkTaxonInfo?taxon_no=34738
https://paleobiodb.org/classic/checkTaxonInfo?taxon_no=64541
Can you fix/redirect these, please?
Big Blue Cray(fish) Twins (talk) 12:20, 12 November 2024 (UTC)
paleodb.org
- Enwiki
- Checked 2,000 pages and edited 1,990 pages. Moved 1,957 links to a new URL: 1,957 ruled soft-redirects, Removed 14
{{dead link}}. Added 9{{dead link}}. Switched 71|url-status=deadto live. Added 30 archive URLs (29 Wayback). Changed 22 citation metadata.
- Checked 2,000 pages and edited 1,990 pages. Moved 1,957 links to a new URL: 1,957 ruled soft-redirects, Removed 14
- Done -- GreenC 02:24, 19 November 2024 (UTC)
fossilworks.org
Big Blue Cray(fish) Twins: From Midshipman fish, there are a lot like this but I couldn't find an equivalent at paleobiodb -- GreenC 05:34, 18 November 2024 (UTC)
- Manually fixed that one. For some reason, they don't match the standard profile, but do still retain the same numbers:
http://www.fossilworks.org/cgi-bin/bridge.pl?a=collectionSearch&collection_no=135043
- became
https://paleobiodb.org/classic/displayCollResults?collection_no=col:135043
- and:
becamehttp://www.fossilworks.org/cgi-bin/bridge.pl?a=taxonInfo&taxon_no=361425
https://paleobiodb.org/classic/basicTaxonInfo?taxon_no=txn:361425
- Thanks for your expert ministrations, but I am afraid I have given you/your bot thousands more!!
- Big Blue Cray(fish) Twins (talk) 09:20, 18 November 2024 (UTC)
- Thank you. Done in Pass 2. More varieties on the margins, if they exist:
- Bohío Formation: displayStrata (603)
- Ashorocetus: displayReference (8)
- Serra da Galga Formation: collectionSearch (156)
- displayStrata has most instances. -- GreenC 15:49, 18 November 2024 (UTC)
- Looks like displayStrata is: http://www.fossilworks.org/cgi-bin/bridge.pl?action=displayStrata&geological_group=&formation=Bohio&group_formation_member=Bohio ==> https://paleobiodb.org/classic/displayStrata?geological_group=&formation=Bohio&group_formation_member=Bohio
- I'll rerun a Pass 3 with this update -- GreenC 21:49, 18 November 2024 (UTC)
- The References are: http://www.fossilworks.org/cgi-bin/bridge.pl?a=displayReference&reference_no=12130 ==> https://paleobiodb.org/classic/displayRefResults?reference_no=ref:12130
- Big Blue Cray(fish) Twins (talk) 22:24, 18 November 2024 (UTC)
- And collectionSearch may be: http://www.fossilworks.org/cgi-bin/bridge.pl?action=collectionSearch&geological_group=Bauru&formation=Mar%EDlia ==> https://paleobiodb.org/classic/displayCollResults?&geologicalgroup=Bauru&formation=Marília
- BUT will need checking against other results to be sure due to Unicode clouding the issue on the example provided
- Big Blue Cray(fish) Twins (talk) 23:34, 18 November 2024 (UTC)
- Running Pass 4 with the new rules, and a larger set of articles. -- GreenC 03:25, 19 November 2024 (UTC)
- Thank you. Done in Pass 2. More varieties on the margins, if they exist:
- Enwiki
- * Pass 1: Checked 7,269 pages and edited 6,391 pages. Moved 3,089 links to a new URL: 3,089 ruled soft-redirects, Removed 17
{{dead link}}. Switched 678|url-status=deadto live. Added 6 archive URLs (6 Wayback). Changed 186 citation metadata. - * Pass 2: Checked 590 pages and edited 525 pages. Moved 1,645 links to a new URL: 1,645 ruled soft-redirects, Removed 2
{{dead link}}. Added 4{{dead link}}. Switched 5|url-status=deadto live. Added 2 archive URLs (2 Wayback). - * Pass 3: Checked 590 pages and edited 67 pages. Moved 603 links to a new URL: 603 ruled soft-redirects
- * Pass 4: Checked 914 pages and edited 423 pages. Moved 687 links to a new URL. Added 20 archive URLs (20 Wayback).
Done -- GreenC 04:41, 19 November 2024 (UTC)
avclub.com
Dead sub-domains. Can be made live again by converting hostname to "www." .. the hostname might be: origin|games|music|film|news|aux|tv|mobile .. 4,732 pages -- GreenC 21:22, 13 November 2024 (UTC)
Enwiki
- Checked 4,742 pages and edited 4,546 pages. Moved 5,181 links to a new URL: 5,156 ruled soft-redirects, 25 ghost soft-redirects, Removed 3
{{dead link}}. Switched 278|url-status=deadto live. Added 31 archive URLs (22 Wayback). Changed 60 citation metadata.
IABot DB
- Checked and updated about 3,500 links that propagate to 300+ wikis
Done -- GreenC 14:55, 19 November 2024 (UTC)
nztop40.co.nz
I'm reposting a request I made at WP:BOTREQ and was directed here.
Dead citations occur due to the the website changing the URL format. For example https://nztop40.co.nz/chart/albums?chart=3467 is now https://aotearoamusiccharts.co.nz/archive/albums/1991-08-09.
Case 1: 9,025 pages that are using these URLs found through search. Some may already be archived.
Case 2: 4,133 citations using {{cite certification
An ideal transition seems difficult as it would require the following steps:
- Find an archived version through the wayback machine, e.g., https://web.archive.org/web/20240713231341/https://nztop40.co.nz/chart/albums?chart=3467 for the above. For case 2 this requires inferring the URL first (
https://nztop40.co.nz/chart/{{#switch:{{{type|}}}|album={{#if:{{{domestic|}}}|nzalbums|albums}}|compilation=compilations|single={{#if:{{{domestic|}}}|nzsingles|singles}}}}?chart={{{id|}}})) - Harvest the date 11 August 1991 either from the rendered archived page or from the archived page source,
<p id="p_calendar_heading">11 August 1991</p> - For case 1, translate the URL accordingly to https://aotearoamusiccharts.co.nz/archive/albums/1991-08-11.
- For case 2, add
|source=newchartand replace|id=1991-08-11.
Note that for case 1, the word after "/archive/" changed according to the following incomplete table. For case 2 this is handled by the template so no need to worry about it.
| Old text | New text |
|---|---|
| albums | albums |
| singles | singles |
| nzalbums | aotearoa-albums |
| nzsingles | aotearoa-singles |
| tereosingles | te-reo-singles |
| hotsingles | hot-singles |
| hotnzsingles | hot-aotearoa-singles |
If someone is willing to go through the above, at least for simple cases, I think it is the ideal solution, especially for case 2. Failing that, a simpler archiving procedure can be taken.
- For case 1: add
|archive-url=and|archive-date=per usual archiving procedure. Add|url-status=deviated. If no archive exists (which should be a minority), add {{dead link}} - For case 2: add
|archive-url=and|archive-date=per usual archiving procedure as they are supported by the templates. Add|source=oldchart(even if no archive is found)
I will be happy to support any technical assistance. Muhandes (talk) 22:55, 14 November 2024 (UTC)
- Muhandes, I don't see any major hurdles with your ideal solution. It's a lot of citations, worth doing. I'm working through requests on this page chronologically. Might get to here in a week or less. -- GreenC 00:51, 15 November 2024 (UTC)
- @GreenC: I'm happy to hear that. In the meanwhile I added records to the table above which should make it complete, to the best of my knowledge. I also noticed some of the URLs (53 of them to be accurate) add an additional #all_records_extra to the URL, e.g., https://nztop40.co.nz/chart/albums?chart=4413#all_records_extra. I will have a look at them individually and perhaps, since it's only 53, do them manually. --Muhandes (talk) 08:18, 15 November 2024 (UTC)
- The pages using #all_records_extra were are all referring to the Heatseeker charts which don't seem to be available on the new website. As such, they should be archived, not translated to the new format. --Muhandes (talk) 10:32, 15 November 2024 (UTC)
- Case 1 and 2 are different code bases. I have a separate code file for working with external link templates. So I'll initially focus on case 1, then likely some of that code can be reused with case 2. -- GreenC 15:07, 19 November 2024 (UTC)
- To document an additional variation, the "End of Year" charts, like this, which have new URLs like
https://aotearoamusiccharts.co.nz/archive/annual-{newcode}/{e}-12-31, where{newcode}is in the HTML search on"<h1>Top Selling [name]"where [name] could be Singles, Albums, NZ Singles, NZ Albums, Compilations - then extrapolate from the chart above. The "{e}" is the year taken from<p id="p_calendar_heading">...</p>-- GreenC 21:11, 19 November 2024 (UTC) - Muhandes: I need help translating the "discover" code as here. I tried this but does not work. -- GreenC 21:48, 19 November 2024 (UTC)
- @GreenC: the "discover" charts are the same Heatseeker charts as the #all_records_extra ones. As far as I can tell they are no longer available. The only way to handle it is to find an archive-url. Note that in these cases the oldest archive-url is the best. I have found several cases where a new archive exists but it does not include the chart itself. Muhandes (talk) 23:45, 19 November 2024 (UTC)
- OK. It defaults to oldest. In the end there were only 3 cases. -- GreenC 00:44, 20 November 2024 (UTC)
- @GreenC: the "discover" charts are the same Heatseeker charts as the #all_records_extra ones. As far as I can tell they are no longer available. The only way to handle it is to find an archive-url. Note that in these cases the oldest archive-url is the best. I have found several cases where a new archive exists but it does not include the chart itself. Muhandes (talk) 23:45, 19 November 2024 (UTC)
Enwiki
- Case 1: Checked 8,904 pages and edited 8,870 pages. Moved 17,224 links to a new URL: 17,224 ruled soft-redirects, Removed 1
{{dead link}}. Added 142{{dead link}}. Switched 313|url-status=deadto live. Switched 38|url-status=liveto dead. Added 269 archive URLs (219 Wayback). Changed 8 citation metadata. - Case 2:
Working -- GreenC 02:34, 20 November 2024 (UTC)
IABot DB
- TBD
iassrt.org
judi. see Special:Diff/1257685967. – robertsky (talk) 04:47, 16 November 2024 (UTC)
KC Chiefs Homepage
kcchiefs.com redirects to chiefs.com without having an archive of the articles. There are 386 articles on English Wikipedia linking to kcchiefs.com Elisfkc (talk) 19:38, 17 November 2024 (UTC)
health.gov
Looks like the original site at health.gov was moved to https://odphp.health.gov and a new health.gov was created. Some links might need updating. --Nintendofan885T&Cs apply 19:39, 18 November 2024 (UTC)
- There are 127 pages. Probably most will convert to archive URLs, unless there is a working redirect. -- GreenC 05:20, 19 November 2024 (UTC)