Remove File System Item Unauthorized

I'm running a single server where I allow customer to remote desktop connect to the server, once they connect our program automatically launches, thus, they never see the desktop, I have it pretty locked down so the standard folders in their User directories aren't needed.  I'm using the below command to go through and remove the Favorites folder:

$folder = "$($user.fullname)\Favorites"
    If (Test-Path $folder) {
        Remove-Item $folder -Recurse 

When I run it with the -whatif it reports correctly, however, when I actually try and run it I get the following:

+ CategoryInfo          : PermissionDenied: (C:\Users\10611A\Downloads:DirectoryInfo) [Remove-Item], IOException
+ FullyQualifiedErrorId : RemoveFileSystemItemUnAuthorizedAccess,Microsoft.PowerShell.Commands.RemoveItemCommand
Remove-Item : Cannot remove item C:\Users\10611A\Favorites: You do not have sufficient access rights to perform this operation.
At C:\DailyRoutine\UserFolderDelete.ps1:21 char:9
+         Remove-Item $folder -Recurse #-whatif

Is this not possible, not advised?  No real reason for doing this other than to save some disk space and reduce whole system backups.

Thanks

Kevin