Saturday, November 7, 2015

Uninstalling Exchange Server 2016

This was like a nightmare.

I installed Exchange Server 2016 on a server machine that had other services running. As a result, I could not just reinstall Windows Server 2012 R2 and started over.

After the installation, Exchange Server 2016 refused to cooperate. All of the Exchange web applications would return "invalid user name or password". The command line power shell could not connect to the Exchange Server upon launch: "Access denied". The Exchange Server Toolbox threw a COM error and would not load.

I was logged in as the domain administrator, but apparently that did not matter. I restarted the server, I re-logged in, Exchange Server 2016 would not work no matter what.

When I tried to uninstall Exchange Server 2016, the uninstaller would refuse to proceed because "there were mailboxes in its database".

Exchange Server 2016 Setup program is really a hybrid application with GUI and power shell command line. Instead of providing on-screen GUI user controls to remove existing mailboxes, it provides a list of commands to be run on the power shell command line that you have to launch by yourself.

Since the power shell could not be launched in my case, the instructions on running these commands were useless unless I could fix this mess first.

Apparently the Exchange power shell uses some of the Exchange web applications in IIS to "connect to Exchange". The fact that there were other certificates in IIS before its installation was something that the Exchange Server 2016 Setup just could not comprehend. The fact that IIS 8 is a retard when handling multiple certificates is the main reason that all this were happening. After decades of development, apparently handling a handful of certificates by subject names and alternative subject names was just way too hard for Microsoft.

So I installed Exchange Server 2016 on a new server just to see what the default settings in IIS were supposed to be.

After the comparisons, I was able to fix some of the Exchange web applications and log into the Exchange "ecp" virtual directory.

But I was still not able to log into "owa" to see my inbox. The authentication would work, but what followed was just a blank page. In the event log I saw that the Exchange "OAB" app could not start because it could not find "Microsoft.Exchange.FrontEndHttpProxy.DLL". Its full path was in a shared config file that the "OAB" app used, and the file was actually there. I tried to place it in the GAC, and it did not make any difference. So I removed it from the GAC, and then copied the entire "HttpProxy\bin" directory under "OAB". This time "OAB" found "Microsoft.Exchange.FrontEndHttpProxy.DLL", but in the event log the new error was that it "does not implement IHttpHandlerFactory or IHttpHandler".

Forget it, I was going to uninstall the whole thing. But the power shell was still not working. After comparing it to a different installation, it turned out that it was trying to do a remote access on the local machine:

C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -noexit -command ". 'C:\Program Files\Microsoft\Exchange Server\V15\bin\RemoteExchange.ps1'; Connect-ExchangeServer -auto -ClientApplication:ManagementShell "

The remote access was failing, so it did not work. After switching to a local version, it worked just fine.

At this point I was able to disable the mailboxes that prevented me from uninstalling Exchange Server 2016.

But the nightmare was over just yet. It turned out that the list of commands in the Exchange Server 2016 Setup was incomplete.

It did not mention anything about "Audit log" mailboxes.

Did I mention that "Get-Mailbox" does not show all of the mailboxes? It does not. So "Get-Mailbox" returns nothing, but the uninstaller refused to proceed because "there were still mailboxes in the database".

Finally I found this category of mailboxes by reading the options for "Get-Mailbox".

So I disabled the last mailbox, and uninstalled Exchange Server 2016, finally.

Well, to make the zombie story complete, after completing the uninstall, the server was rebooted, but the event log continued to receive Exchange Server 2012 errors non-stop.

It turned out that the Exchange Server 2016 Setup left behind its "back end web site" in the IIS, but the name changed to just "Site 7" (on other machines this could be "Site 4" or whatever ID it was using). I could not even click on this site because it did not have its files. So I clicked on the root "Site" folder of  IIS, and deleted it from the right pane of the IIS manager.

In addition, it left a bunch of application pools in IIS as well. What was worse was that it also left changes on EVERY application pool in IIS, including those that did not even belong to Exchange Server 2016. These changes were not even visible in the IIS manager. You would have to get into "C:\Windows\System32\inetsrv\config\applicationHost.config" and remove them manually. You would need to go into the "applicationPools" section and remove "CLRConfigFile='C:\Program Files\Microsoft\Exchange Server\V15\bin\GenericAppPoolConfigWithGCServerEnabledFalse.config'" from EVERY application pool. In the "globalModules" section, you would have to remove "".

That cleaned up the server for good.