Print | posted on Monday, September 20, 2010 3:49 PM
Introduction
Back about a week after RTM of SharePoint 2010 I published my Rational Guide to implementing SharePoint Server 2010 User Profile Synchronization. This was actually written up long before RTM and was doing the rounds among a circle of SharePoint “insiders”. I then tweaked it for RTM and pushed it out immediately after the SharePoint Evolutions Conference, where I had demonstrated the steps.
Amazingly, this article has already been viewed over 260,000 times! An incredible response. Of course the Microsoft documentation in this area is weak at present, and UPS is what you could call a “rough edge” of SharePoint 2010. But even so this is an amazing number. Furthermore the amount of time I spend helping others with UPS related problems as a result has inspired this follow up article which will cover the most common issues people experience.
[UPDATE: 01/10/2010] TechNet has recently updated its Configure profile synchronization (SharePoint Server 2010) topic, which is greatly improved and now a first class resource. I urge you to check this out.
Before I dive in, I must stress that the number one reason people have problems is that they do not follow the procedure! No really, I can’t count the number of times steps have been missed or I get a response like “oh, I didn’t think I needed to do that”. If you follow the procedure you will be successful unless you are hitting an environmental or other known issue.
I won’t detail all of the known issues in this article, but rather cover the most common ones. The general idea is to have a reference for common queries and so on. I will update this article over time as “new” things are “discovered”.
Once again I cannot stress how important it is to follow the procedure in the Rational Guide. If you haven’t, go there first!
This article will use the following definitions for the various components involved:
- UPA: The User Profile Service Application (in Manage Service Applications)
- UPS: The User Profile Synchronization Service Instance (in Services on Server)
Why does UPS get “stuck on starting”?
Why does it get stuck? Well, 15 provisioning runs will be attempted before it gives up. When we hit Start in Services in Server a series of tasks are run, which are akin to running the second stage of the Forefront Identity Manager setup. Those include things like setting up certificates, windows services, installing database schemas so on. If something is incorrect, it will try all 15 times, and more often than not, fail 15 times. That’s why it appears to be “stuck” when in fact its attempting each of the runs. Of course this can take a while, hence the appearance of being “stuck”. After around 20 to 45 minutes the status will return to Stopped, and you can attempt to start it again.
In some cases, the status will remain as “Starting” even after reboots. In this case something very bad has happened and you are best off throwing away the UPA service application and starting over. Hacking out a solution is not a good idea, and will almost certainly cause you further problems in the future.
You can however, manually kick things back to the state before the starting of the UPS service instance. This can be necessary in some cases, but again, I strongly recommend you start clean by deleting and recreating the UPA. Steps to reset the state can be found here.
Incorrect Permissions
Incorrect Permissions are the number one cause of the infamous “Stuck on Starting” behaviour. There are no ways around the account requirements. If you don’t set the permissions correctly things won’t work, its that simple.
There are two accounts we need to configure. The UPS Service Instance Service Identity, i.e. the account that the two FIM services run as (which is the Farm Account). And the Synchronization Connection account, this is the account which actually performs the sync.
The UPS Service Instance Service Identity.
- Firstly, you must run the UPS Service Instance as the Farm Account. Just because its possible to change the service identity in Manage Service Accounts, doesn’t mean it will work, and more importantly it is unsupported. So don’t change it.
- Next, the Farm Account must be a local administrator of the machine running the UPS Service Instance during provisioning only. When we hit Start in Services in Server a series of tasks are run, which are akin to running the second stage of the Forefront Identity Manager setup. Many of these tasks require local machine administrator rights. You must grant this right before hitting start, and more importantly they must be applied. As the Farm Account is running services on your box (SPTimerV4 and the Central Admin app pool) we must simulate a log off and log on for the change in rights to be applied. You can do this by restarting SPTimerV4, or better yet, rebooting the machine. Once UPS is provisioned you can remove the Farm Account from the local administrators group.
[Update] Please note that any event in your farm that requires the UPS service instance to be provisioned will require the Farm Account be a local admin. Such events include the re provisioning of the service instance following the deployment of a SharePoint Cumulative Update and performing a Farm Backup from Central Administration (which stops and starts the UPS service instance). Don’t forget to ensure that the correct rights are assigned (and actually taking effect) when planning and scheduling your farm operational maintenance tasks. - Also, the Farm Account must have Log on Locally rights on the machine running the UPS Service Instance. Now be careful, you may think the account has these rights, and it will do if you made it a local administrator. However once you remove the admin rights, you may not have Log On Locally. If you don’t various operations will fail.
The Synchronization Connection Account
- The Synchronization Connection Account must have Replicating Directory Changes on the Domain you are syncing with. It is the only way sync will work. It’s a hard requirement. Despite the scary name the Replicating Directory Changes permission makes zero changes to AD. It provides a change log capability, which improves the speed of operations such as Sync.
- If you are syncing with a Windows Server 2003 functional level domain, the Synchronization Connection Account must be a member of the Pre-Windows 2000 Compatible Access group.
- If you wish to do an Export, the Synchronization Connection account must have Create Child Objects and Write permissions on the OUs you are syncing with. Watch out for a common gotcha, that these rights are set on the container level only, you must ensure that This object and all descendants is selected.
So there you have it, a summary of the rights required. What if you are not the domain administrator? Well before you start, ask for proof in the form of a screenshot that they are set correctly. Make sure you verify these permissions before you start working in SharePoint. Over and over again I’ve seen people waste hours (or days) on this because they weren’t.
It’s also very common to mix up these accounts. Remember the Farm Account runs the service, and the Sync Connection account does the sync. Its not very sensible to use the same account for both.
No accounts ever need Domain Administrator privileges to get UPS working.
You don’t need to log on as the Farm Account ever to get UPS (or anything else) working. Logging on as the Farm Account is a very, very bad idea and you shouldn’t be doing it.
You of course require Farm Administrator rights to perform the actions related to the farm, service application administrator rights to perform actions related to the service application, and local machine administrator to perform actions related to the machine configuration detailed below.
Health Analyzer Warning
Please note, that the Health Analyzer will report an error: “The server farm account should not be used for other services”. This will pop up every week by default. Unfortunately, this Health Analyzer Rule is not smart enough to understand that you have to run UPS as the Farm Account. Therefore this error (with respect to UPS only) should be ignored. You could choose to disable this rule if you are confident that your farm administrators won’t use the farm account elsewhere. One hopes that this will be modified in a future update.
Configure Service Accounts
Whilst it is possible to change the UPS account using Configure Service Accounts, this is NOT supported and it will not work properly. You may think it’s working, but you cannot have 100% fidelity UPS under any account other than the Farm Account. It is a bug that UPS shows up in this screen.
If you can’t get the UPS provisioning to even kick in and attempt to work, its likely you have run the Farm Configuration Wizard (FCW) and are attempting to start the UPS Service Instance associated with the User Profile Service Application created by the FCW.
Whilst it is possible to get UPS working in this configuration, you shouldn’t be doing it, and I’m not going to detail the steps.
Don’t use the Farm Configuration Wizard! That’s all I have to say on this issue.
Note: If you want the low down on what to do next if you have used the FCW, my fellow MVP, Woody Windischman, has some details over on The Sanity Point.
If the NetBIOS domain name and it’s fully qualified name do not match there is additional configuration necessary. This does not effect provisioning, but it will prevent sync from working. You must do the steps below in the correct order, otherwise you will encounter problems with the SyncDB. Do them in the correct order!
Additional Permissions (Do this first)
- The Synchronization Connection account must have Replicating Directory Changes on the cn=Configuration naming context. You can also perform this using the Advanced Features view of ADUC if you wish.
- Start… Run… ADSIEdit.msc
- Connect to the Configuration Partition
- Right click the configuration partition and choose properties
- From the Security tab, add the Synchronization Connection account and give it Replicating Directory Changes permissions
Configure the User Profile Service Application to support NetBIOS names
Now we can go ahead and provision UPS and configure our Synchronization Connections.
[UPDATE]
Note: the December 2010 Cumulative Update breaks this capability and after setting NetBIOSDomainNamesEnabled, you will not be able to create Synchronization Connections. If you need this capability, do not install the December 2010 CU!
This issue is resolved in the February 2011 CU. Once you have applied the CU and then set the property of the UPA, perform an IIS Reset before attempting to create sync connections.
If you are hosting your SharePoint databases on a SQL Server Named Instance there is additional configuration required.
The SharePoint Configuration Wizard (PSConfig) supports named instances (e.g. SQLServerName\InstanceName), but UPS does not. Therefore if we wish to use named instances and UPS we must configure an Alias, and we must do this before we run PSConfig to create the farm.
Note: this issue is fixed in the June 2010 Cumulative Update for SharePoint Server. Thus if we are running the June CU or later, we do not need to configure an alias. However, I strongly recommend you use an alias, as it is a best practice for addressing SQL named instances regardless of this issue.
To do this we should run the SQL Server Client Network Utility (which is installed on every SharePoint machine).
- Start… Run..
- Type cliconfg and click OK.
- Click TCP/IP and then the Enable >> button.
- Click the Alias tab.
- Click the Add button.
- Select the TCP/IP radio button.
- Enter the alias you wish to use (e.g. SHAREPOINT) in the Server alias text box.
- Enter the address of your instance (e.g. SQL1\SHAREPOINT) in the Server name text box.
- Deselect the Dynamically determine port check box.
- Enter the port of your instance (e.g. 1433) in the Port number text box.
- Click OK to save the alias.
- Click OK to save the configuration and close SQL Server Client Network Utility.
Once we have an alias we can create our farm using it. However there is also another step necessary for reliable start up of the UPS service instance. Basically what happens is that we can provision UPS, but when we restart the machine (for example after patching the box) the UPS services will fail to start. We should configure this before starting the UPS service instance for the first time to avoid the issue completely.
We need to open up network access to the Local DTC on the machine hosting the UPS Service Instance, which is done using the Component Services MMC Snap In:
- Start… Administrative Tools… Component Services.
- Expand Component Services > Computers > My Computer > Distributed Transaction Coordinator.
- Right click Local DTC and choose Properties.
- Click the Security tab.
- Check the Network DTC Access check box and the Allow Remote Clients check box.
- Click OK.
- You will be prompted to restart MSDTC, click Yes.
Now we can provision UPS and it will start reliably following a machine restart.
Note: it is common to see an Event in the Application Event Log complaining about lack of Local DTC Network Access during UPS provisioning. However the only time you need to alter this configuration is if you are using a SQL Server Named Instance.
If you have created the User Profile Service Application (UPA) using PowerShell there is a bug in the creation of the Sync DB which will prevent the UPS provisioning from succeeding. Before we start UPS we must fix this up.
The issue is that the default schema for the Farm Account in the Sync DB is set incorrectly. When UPS provisioning attempts to install the schema it will fail. This is one of the times we MUST touch the SharePoint databases. At present this is the ONLY way to get this working.
We simply need to change the Default Schema of the Farm Account to be DBO after we create the User Profile Service Application and before we start the UPS Service Instance. We do not need to make the Farm Account a DBO.
Please note that altering the Database in this fashion is unsupported. Another workaround here is to create the UPA from Windows PowerShell whilst logged on interactively as the Farm Account using UAC elevation. This effectively does the same thing as creating the UPA with Central Administration (which of course runs under the Farm Account).
So you have a choice about how to fix this issue. Either run the Windows PowerShell while logged on as the Farm Account, or alter the database user default schema. Whilst the database approach is unsupported, logging on as the Farm Account is a very bad idea, and something you shouldn’t be doing. If you do this to avoid this issue cool, but you shouldn’t be doing it generally. Of course the logon as Farm Account approach has serious implications for any automated build scripts you may be pursuing.
An alternate to the log on as the farm account approach is to make use of the Get-Credential and Start-Job Cmdlets when creating the User Profile Service Application. This allows us to avoid the default schema problem without logging on interactively or manually touching the database. It’s not ideal, but it is a fully supported approach, and it works!
$sb = {
Add-PSSnapin Microsoft.SharePoint.PowerShell
$saAppPool = Get-SPServiceApplicationPool "SharePoint Web Services Default"
$upa = New-SPProfileServiceApplication -Name "User Profile Service Application" `
-ApplicationPool $saAppPool `
-ProfileDBName "UPA1_Profile" `
-SocialDBName "UPA1_Social" `
-ProfileSyncDBName "UPA1_Sync" `
-ErrorAction SilentlyContinue -ErrorVariable er
}
$cred = Get-Credential "sharepoint\spfarm"
$job = Start-Job -Credential $cred -ScriptBlock $sb | Wait-Job
Please see the post
for more details.
If you have already created the User Profile Service Application, you either have to throw it away and start over using the above, or manually fix up the default schema for the farm account. Note that this is NOT supported.
- Connect to your SQL instance using SQL Server Management Studio
- Expand the Databases node.
- Expand the Sync DB and the Security and Users nodes.
- Right click on the Farm Account user and choose Properties.
- Click the ellipses (…) to the right of the Default Schema text box.
- Enter dbo and click OK.
- Click OK to save the change and close SQL Server Management Studio.
We can also make this change using a simple T-SQL statement:
ALTER USER [SHAREPOINT\spfarm] WITH DEFAULT_SCHEMA=dbo
We can also use the SQL Server PowerShell snapins to do this:
$upaSyncDBName = "SyncDBName" Invoke-sqlcmd -Database $upaSyncDBName -Query "ALTER USER [SHAREPOINT\spfarm] WITH DEFAULT_SCHEMA=dbo;"
Once we have fixed up this problem, we can go ahead and provision the UPS Service Instance.
If you are running your SharePoint Central Administration over SSL (which you should be in production) on the default zone, profile synchronization will fail and you will notice events 6801 in your event log. This is a bug where the core FIM Management Agent is configured with incorrect connection information. Actually it’s not incorrect, but it doesn’t account for a SSL connection.
Note: this issue is fixed in the October 2010 Cumulative Update and later.
The following steps are for those who cannot move to the October 2010 CU or later. This approach is NOT supported.
To fix this we must manually edit the Management Agent Connection Information. We must do this after UPS provisioning.
- Start… Run…
- Type c:\Program Files\Microsoft Office Servers\14.0\Synchronization Service\UIShell\miisclient.exe and hit OK.
- Click the Management Agents tab.
- Right click the management agent who's Name starts with “MOSS” and is followed by a GUID, and choose Properties…
- In the Management Agent Designer click Configure Connection Information.
- In the Connection Information area, within the Connect To: text box, replace the protocol identifier (direct://) with https:// and also ensure that it includes the port (:443).
- Click OK to save the changes, and exit the Synchronization Service Manager.
Now profile synchronization will work as expected.
Note: the use of the Synchronization Service Manager (miisclient.exe) to make configuration changes is not supported by Microsoft. Deploy the October 2010 CU or later to resolve this issue.
Timeouts when creating Synchronization Connections
When you attempt to Populate Containers or save a Synchronization Connection you may receive a client side timeout error. This can be due to a large number of containers in the domain, or other domain connectivity issues. We can adjust the timeouts used using Windows PowerShell.
Firstly, the Populate Containers timeout, which by default is 30 seconds. We set this property on the User Profile Service Application Proxy:
$upaProxy = Get-SPServiceApplicationProxy -Id <GUID of User Profile Service Application Proxy>
$upaProxy.ImportConnAsyncTimeout = 45
$upapProxy.Update()
# To get the GUID of the User Profile Service Application Proxy run Get-SPServiceApplicationProxy
Next, the Save Synchronization Connection timeout, which by default is approximately 17 minutes. We can adjust this value (in milliseconds this time) on the Service Application:
$upsa = Get-SPServiceApplication –Id <GUID of User Profile Service Application>
$upsa.FIMWebClientTimeOut = 240000
$upsa.Update()
# To get the GUID of the User Profile Service Application run Get-SPServiceApplication.
Lastly you may receive timeouts when simply connecting to the domain. By default the maximum time is 30 seconds. To alter this value, we must install the June Cumulative Update or later. Once we have done that we can modify the connection timeout on the Proxy:
$upaProxy = Get-SPServiceApplicationProxy -Id <GUID of User Profile Service Application Proxy>
$upaProxy.LdapConnectionTimeout = 45
$upapProxy.Update()
# To get the GUID of the User Profile Service Application Proxy run Get-SPServiceApplicationProxy
I strongly recommend that you slowly increase the timeouts until they work, rather than simply entering a very big number! : )
After a server restart, the UPS service instance is stopped
If you are hosting everything (Active Directory, SQL Server, SharePoint) on a single server Virtual Machine, it is very common for the UPS service instance to be stopped after a restart of the machine. This is commonly because the two FIM services need SQL Server up and running and responding to connections. If SQL Server takes along time to come up as is often the case on a domain controller, the FIM Synchronization service will fail to start.
You can avoid this issue by changing the start-up behaviour of the FIM Synchronization service to “Delayed Start” within Services.msc. You could configure the FIM Synchronization service to have a dependency on SQL Server, but I don’t recommend that. Of course for production environments you shouldn’t be making either change, but then for production you should never host AD and SQL and SharePoint on the same machine!
After the installation of Cumulative Updates, the UPS service instance needs to be re-provisioned.
After the installation of Cumulative Updates for SharePoint 2010, the UPS service instance will require re-provisioning, and it’s state will be stopped. This behaviour is by design. When CUs are applied, we run PSConfig(UI), which amongst other things will upgrade the schema of the SharePoint databases. However PSConfig doesn’t touch the Sync DB. Hence we need to re-provision UPS to update any changes to it’s schema. Re-provisioning is significantly quicker than the initial provisioning but does require the same rights.
Re-provisioning the UPS service instance does not affect the User Profile Service Application (UPA) it simply means repeat the process of Starting the UPS Service Instance.
When attempting to Manage the UPA, you receive an error, Not Found, Correlation ID: [guid], or Could not load file or assembly 'Microsoft.ResourceManagement’
This can occur once you have installed the June 2010 Cumulative Update (CU) for SharePoint Server 2010. The June CU actually requires installation of two different hotfixes (in addition to the SPF June CU). You must install both of these. If you install just KB983497, you will receive the above errors.
I strongly recommend you deploy the August 2010 CUs instead, which offer a much better installation experience.
After the installation of the August CU, Event ID 3 is repeatedly logged to the Application Event Log
This is a known issue with the August 2010 CU, or later, which is due to a version mismatch in the Sync DB. The error can be safely ignored, and UPS will continue to work as expected. However there are cases where this mismatch causes operational problems. This error does not occur if you install the August 2010 CU or later before provisioning the UPA Service Application and UPS service instance.
If you want to get rid of this message you can also delete and re-create the UPA service application. This is a pretty drastic action for an erroneous error! and you will need to consider the data you are throwing away in this procedure carefully. I do not recommend this course of action. Whilst you can do this and retain data, its not clean. If you can, wait for a future fix which will resolve the version mismatch and therefore the erroneous event log entry.
However, this is not a fix that can be easily delivered by PSConfig. As mentioned above in the section about CUs, PSConfig does not touch the Sync DB, and it’s the provisioning of UPS that fixes up the mismatch, as you can see below:
Therefore don’t expect a fix for this. Get your UPA in overall good shape now, don’t wait thinking this will be sorted in the near term.
After clicking Start in Services on Server, nothing happens for 20 minutes and UPS is not provisioned. No ILM Configuration steps are logged to the ULS
This condition occurs when a Fully Qualified Name (FQDN), for example, SQLSERVER1.domain.com, or IP Address has been used as the address for the SQL Server in the SharePoint Configuration Wizard (PSConfig) when creating the Farm. UPS (amongst other things) has major issues with FQDNs and IP addresses. Don’t use them! Use a NetBIOS name (e.g. SQLSERVER1) or a SQL Server Alias when creating the farm.
How to view progress of UPS Provisioning
The best way to view progress of UPS provisioning, and identify the root cause of any failures is to examine the SharePoint ULS logs. All events related to the provisioning of the UPS service instance are logged with a Category of User Profiles. Therefore we can filter the logs for these entries. I strongly recommend ULSViewer which allows you to view events in real time, alongside a host of other features. ULSViewer is the single most important tool in the box for a SharePoint practitioner.
If you are hitting problems and contact me for help, I will ask you for the ULS.
Here you can see the events which take place after hitting Start in Services on Server for the UPS Service Instance in ULSViewer, you can click the image for the full size.
A successful provisioning, as you can see above, takes around three minutes.
Any errors will be highlighted in here which will point you to the appropriate troubleshooting steps above, which is more often than not (over 75% of all queries I have got), incorrect permissions!
In some cases, the UPS status will remain as “Starting” even after reboots. In this case something very bad has happened and you are best off throwing away the UPA service application and starting over. Hacking out a solution is not a good idea, and will almost certainly cause you further problems in the future.
You can however, manually kick things back to the state before the starting of the UPS service instance. This can be necessary in some cases, for example if you have provisioned the UPA with Windows PowerShell, but not set the correct default schema for the Farm account before attempting to start UPS.
I strongly recommend you start clean by deleting and recreating the UPA.
The following Windows PowerShell can be used to reset the Sync Machine Instance:
Add-PSSnapin Microsoft.SharePoint.Powershell
# these will only work if there is one DB and one SA on the box.
# If more than one, then use Get-SPDatabase and Get-SPServiceApplication
# to grab the GUID and pass that in instead of the pipebind
$syncDBType = "Microsoft.Office.Server.Administration.SynchronizationDatabase"
$upaSAType = "User Profile Service Application"
$syncDB = Get-SPDatabase | where-object {$_.Type -eq $syncDBType}
$upa = Get-SPServiceApplication | where-object {$_.TypeName -eq $upaSAType}
$syncDB.Unprovision()
$syncDB.Status = "Offline"
$upa.ResetSynchronizationMachine()
$upa.ResetSynchronizationDatabase()
$syncDB.Provision()
# we MUST restart the timer service for the state to be reflected in
# Services on Server and Manage UPA
restart-service SPTimerV4
# at this stage we MUST add the Farm account to the SyncDB (the above
# steps remove the user) remember the default schema must be 'dbo'.
# If we don't do this, UPS provisioning will fail.
This is due to insufficient privileges for the SharePoint Farm Account on the Sync DB (not the Config DB!). You need to add the farm account to the Sync DB users as DBO with a default schema of DBO and then start UPS again.
After installing the December 2010 CU and attempting to create a Synchronization Connection with multiple domains you will receive an "Unable to process create message" error.
This issue is resolved in the February 2011 CU.
If you cannot deploy the February 2011 CU you need to first create a Sync Connection with a single domain, and then edit it to add the additional domains. Please see KB2490381 for more details.
After installing the June 2011 CU the User Profile Synchronization service instance will show as Stopped within Services on Server. Additionally while the Forefront Identity Manager service will be started in Services.msc, the Forefront Identity Manager Synchronization service will be stopped.
You must restart the UPS service instance from Services on Server after installing the June 2011 CU. Remember to ensure that the Farm Account is a local administrator on the machine running UPS!
Even if the UPS service instance is still started after deploying the June 2011 CU it must be restarted before successful sync operations can be performed.
You must restart the UPS service instance from Services on Server after installing the June 2011 CU. Remember to ensure that the Farm Account is a local administrator on the machine running UPS!
Profile Synchronization fails with Stopped Extension DLL errors reported in the Synchronization Service Manager (miisclient.exe) and System.PlatformNotSupported exceptions are written to the Event Log. This is due to a change made to the FIM config files.
Remove the following lines from %PROGRAMFILES% \Microsoft Office Servers\14.0\Synchronization Service\bin\miiserver.exe.config
<supportedRuntime version="v4.0.XXXXX"></supportedRuntime>
Remove the following lines from %PROGRAMFILES% \Microsoft Office Servers\14.0\Synchronization Service\bin\mmsscrpt.exe.config
<supportedRuntime version="v4.0.XXXXX"></supportedRuntime>
Restart the FIM and FIM Sync services using Services.msc or the command prompt
net stop FIMService
net start FIMService
net stop FIMSynchronizationService
net start FIMSynchronizationService
Or better yet, using Windows PowerShell
Restart-Service FIMService
Restart-Service FIMSynchronizationService
[UPDATE: 12/07/2011] The June 2011 CU was “re-released” today. If you deploy the new version of this CU, the problem does not occur.
Conclusion
There we have it. Details of some common problems with the User Profile Synchronization capability in SharePoint Server 2010, which I hope will be useful. I will be updating this article from time to time in the future, hopefully by adding “install this patch” : ) but also as new issues come to light.
.