If you’ve built a custom sign in page in SharePoint 2010 but keep getting a 401 Unauthorized error instead of the sign in page then you’ve most probably deployed the solution package to a single web application.
To resolve the issue the solution needs to be deployed to all applications and not restricted to a single web application.
i.e. When calling Install-SPSolution don’t include the -WebApplication parameter.
Edit: Another instance when you will see the 401 Unauthorized error is when your not inheriting from the correct base page. It should not be the layoutsbasepage.
Another tip if you see the following error in your ULS log:
Request for security token failed with exception: System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail]: Object reference not set to an instance of an object. (Fault Detail is equal to An ExceptionDetail, likely created by IncludeExceptionDetailInFaults=true, whose value is: System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.SharePoint.IdentityModel.SPSecurityTokenService.SPRequestInfo.ValidateFormsAuthProviderNames(Uri context, String membershipProvider, String roleProvider)
at Microsoft.SharePoint.IdentityModel.SPSecurityTokenService.SPRequestInfo.SetProviderNames(RequestSecurityToken request)
at Microsoft.SharePoint.IdentityModel.SPSecurityTokenService.SPRequestInfo..ctor(IClaimsIdentity identity, RequestSecurityToken request, Boolean initializeForActor)
at Microsoft.SharePoint.IdentityModel.SPSecurityTokenService.SPRequestInfo..ctor(IClaimsPrincipal principal, RequestSecurityToken request)
at Microsoft.SharePoint.IdentityModel.SPSecurityTokenService.GetTokenLifetime(Lifetime requestLifetime)
at Microsoft.IdentityModel.SecurityTokenService.SecurityTokenService.Issue(IClaimsPrincipal principal, RequestSecurityToken r...).
That means your using a name that has not been setup as an AAM. I was using http://localhost and it always kept failing. Changing the request url to the actual name of the server (in my case http://mosswfe1) fixed it.
Note to myself: References for creating a Custom Sign In Page
Creating a Custom Login Page for FBA in SharePoint 2010
Creating a Custom Login Page for SharePoint 2010
How to create custom login page for SharePoint 2010 form based authentication (FBA)