I had a few admin PHP pages in a custom subfolder for a customer, protected with the AuthUser directive in the .htaccess file. However, when I migrated their main site from Zen Cart to Magento (moving the Magento folder to the root), the subfolder kept throwing 404 errors from Magento.
I knew which line was doing it, but the final redirect should have been ignoring real files and folders:
RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteCond %{REQUEST_FILENAME} !-l RewriteRule .* index.php [L]
Finally I stumbled across the answer, which was not on the main .htaccess folder. In the .htaccess file in the protected subfolder, I needed to add a single line, which probably should have been there the whole time:
ErrorDocument 401 "Unauthorized Access"
The problem was fixed!