ok first. make sure your webconfig file is similar to the one below.

Make sure on IIS that the only authentication scheme you have selected is
Windows authentication for you application and the site it is contained withing. This is because WCF and RIA doesn't support multiple authentication schemes within the same site.
Next for your application in IIS
Make sure that the default document is setup to point to the aspx page that your application created when published. Once that is setup browse to your site like this. http://Sitename/Application name if you don't browse to the site like that you might receive the same error. I had hell getting this setup and I hope this makes it easier for other users that have came across this same issue.
Please feel free to leave comments or add to this Post other steps that can deviate from the frustration in deploying your new silverlight app.