Force DNS Lookup Over Windows VPN Connection

A common problem with Remote Access VPN connections is how to force DNS name resolution across the VPN tunnel. With Windows you will likely need to modify the Windows DNS Client behavior by changing the Network Interface Binding Order (older versions of Windows), or Network Interface Metric (Windows 10).

According to the Microsoft Networking Support Team, “The only known component that used the binding order was DNS ordering. By default, Windows uses the Route Metric + Interface Metric to determine which route has the highest priority by choosing the route with the lowest value.” Because of this, you may need to adjust the interface metric(s) using GUI, PowerShell (Set-NetIPInterface -InterfaceMetric), or netsh (int ip set inerface=”…” metric=…) to ensure that the VPN connection(s) automatically assigned metric will be lower (preferred) over any wired or wireless connection you may be using to access the Internet on your computer.

For reference, here are two StackExchange Q&A discussions of the problem and solution options.

• How does Windows decides which DNS Server to use when resolving names? (serverfault.com)
• Windows 10 DNS resolution via VPN connection not working (superuser.com)

An important note is that when adjusting the interface metric(s) on Windows 10, you should only adjust the metric on NON-VPN interfaces since the VPN interface metric will be reset automatically every time you connect. Base the setting of your other interface metric on the automatic value you see assigned to your VPN connection so that the VPN will always be preferred (lower metric).

Good luck with your Windows VPN DNS troubleshooting and repair tasking! 🙂

View Certificates Issued to Hostname in ADCS

Active Directory Certificate Services (ADCS) is a useful service to provide in a Windows Domain environment. If you’re managing ADCS, you might want a quick way to list issued certificates to a specific hostname. Here’s a quick PowerShell script that helps put the right filters on “certutil -view” command to list certs for a given computer name.

	# Change the filter and column list as needed to match your query needs.
	# Query Active Directory Certificate Services for Certs issued to given hostname
	function view_certs($prefix, $config=$null) {
	# call like: view_certs -prefix "srv-name"
	# Find $config value by running "certutil" with NO OPTIONS
	# increment last character to get next prefix (stop matching)
	$nextprefix = $prefix.Remove($prefix.Length-1) + [char]([int]$prefix[-1] + 1)
	$columns = "CommonName,DispositionMessage,CertificateTemplate,NotBefore,NotAfter"
	# Disposition of 20 returns only "Issued" certificates
	$restriction = "CommonName >= $prefix,CommonName < $nextprefix,Disposition = 20"
	if ($config -ne $null) {
	certutil.exe -view -config "$config" -restrict "$restriction" -out "$columns"
	} else {
	certutil.exe -view -restrict "$restriction" -out "$columns"
	}
	}
	# example calling function for hostname beginning with prefix
	view_certs -prefix "srv-name"
	# Inspired by
	# … https://blogs.technet.microsoft.com/pki/2008/10/03/disposition-values-for-certutil-view-restrict-and-some-creative-samples/

view raw

view_certs.ps1

hosted with ❤ by GitHub

Code search keywords: certutil, view, restrict, CommonName, Disposition, DispositionMessage, CertificateTemplate, NotBefore, NotAfter

This was inspired by the following:

• Disposition values for certutil –view –restrict (and some creative samples)

Clear adminCount and Enable Inheritance on User

Users in Active Directory that belong to factory default privileged groups will be protected by an hourly process called SDProp (Security Descriptor Propagator) on the Primary Domain Controller (PDC Emulator). This process sets the adminCount attribute to 1 and disables inheritance on the user object in the directory. It also applies access control list (ACL) rules to the object from the AdminSDHolder template defined in the directory.

If you want to re-enable permissions inheritance that has been protected by SDProp / AdminSDHolder, you will need to do something like the following.

• Remove the user from all protected groups (factory default privileged groups in the directory like Domain Admins, Server Operators, Print Operators, Account Operators, Backup Operators, etc. NOTE that protected group membership can be inherited when one group is a member of another nested group.
• Clear the adminCount property on the User object (either set to 0, or unset the property completely)
• Re-enable inheritance on the User object
• Wait an hour or two and then verify that the adminCount and blocked inheritance are not later re-applied by the domain SDProp service.

Here is a script in PowerShell that will take a CSV of usernames and clear adminCount plus re-enable inheritance on each user object for you. It will also report on the groups these user accounts belong to – search it for any protected groups.

	$ErrorActionPreference = "Stop"
	# Put usernames one-per-line in a text (CSV) file with header first-line: Username
	# Set this to the path of your input csv file.
	$CSVFilePath = "TroubleUsers.csv"
	# This script will Clear the adminCount property and Enable Inheritance on the object Access Control List (ACL) in AD
	$users = import-csv $CSVFilePath | sort Username
	$groups = @()
	foreach ($user in $users) {
	$aduser = get-aduser $user.Username -properties ntSecurityDescriptor,adminCount
	$aduser | select Name,SamAccountName,adminCount,
	@{Name="AreAccessRulesProtected";Expression={$_.ntSecurityDescriptor.AreAccessRulesProtected}}
	$adgroups = Get-ADPrincipalGroupMembership $aduser
	foreach ($group in $adgroups) {
	$groupname = $group.SamAccountName
	if ($groups -notcontains $groupname) {
	$groups += $groupname # add groupname to list
	}
	}
	if ($aduser.ntSecurityDescriptor.AreAccessRulesProtected -or
	($aduser.adminCount -ne $null -and $aduser.adminCount -gt 0)) {
	# update security descriptor prior to Set command (ENABLE INHERITANCE)
	# … SetAccessRuleProtection(bool isProtected, bool preserveInheritance)
	$aduser.ntSecurityDescriptor.SetAccessRuleProtection($false, $true)
	Set-ADUser -Identity $aduser -Replace @{ntSecurityDescriptor = $aduser.ntSecurityDescriptor} `
	-Clear "adminCount" -verbose
	}
	}
	"Look for Protected Groups – these will cause hourly SDProp to re-apply object protection (AdminSDHolder)"
	$groups | sort
	# Inspired by the following articles:
	# https://blogs.msdn.microsoft.com/muaddib/2013/12/30/how-to-modify-security-inheritance-on-active-directory-objects-using-powershell/
	# https://social.technet.microsoft.com/Forums/office/en-US/ed7e5c9f-b455-4878-8970-16e362717245/ad-user-update-inheritable-persmission

view raw

AdminSDHolderFix.ps1

hosted with ❤ by GitHub

Code search keywords: Get-AdUser, ntSecurityDescriptor, ObjectSecurity.SetAccessRuleProtection, ObjectSecurity.AreAccessRulesProtected, Get-ADPrincipalGroupMembership, SamAccountName, adminCount, Set-AdUser, Replace, Clear

This post has been inspired by the following:

• Delegated permissions are not available and inheritance is automatically disabled (vbscript sample under section “Method 1: Make sure members are not members of a protected group”)
• How to Modify Security Inheritance on Active Directory Objects using PowerShell (blogs.msdn.microsoft.com)
• AD User – Update inheritable persmission (TechNet Windows PowerShell Forum)

Cisco ASA Defaults that Break Things

The Cisco ASA is an excellent network security device, but several factory defaults can break some typical network traffic. These settings might be good to change on standard deployments.

• Turn On “inspect icmp” to allow ping responses through
• Turn On “inspect icmp error” to fixup non-ping ICMP responses through NAT
  • Enable incoming icmp time-exceed and unreachable traffic on outside incoming access-list
  • This allows return messages for path mtu detection (PMTUD) and traceroute among other important things used by the TCP/IP stack to deal with end-to-end MTU limits and other common routing errors.
• Turn Off “inspect esmtp” under global_policy inspection_default
  • This setting often breaks communication between mail servers like email gateways and Exchange servers. It also breaks the ability to test SMTP with telnet or other common troubleshooting tools.

There are many settings out-of-box that a deployment might want to change, but these are a few that have been on my mind recently and I wanted to write this down. 🙂

View Server Certificate with PowerShell

This is a follow-up to my previous post on checking Server Certificate detail with OpenSSL. Note that this example lacks error checking – that is left as an exercise for the reader. This is based on the “Parallel Universe” blog post “Reading a Certificate off a remote SSL Server for Troubleshooting with Powershell!” – many thanks to John McCabe for sharing.

	# This example lacks error checking
	# late binding, call main AFTER defining check_ssl dependency
	function main() { check_ssl -fqdn "your.fqdn.com" -port 443 }
	function check_ssl($fqdn, $port) {
	$tcpsocket = New-Object Net.Sockets.TcpClient($fqdn, $port)
	$tcpstream = $tcpsocket.GetStream()
	$sslStream = New-Object Net.Security.SslStream($tcpstream, $false)
	$sslStream.AuthenticateAsClient($fqdn)
	$certinfo = New-Object security.cryptography.x509certificates.x509certificate2($sslStream.RemoteCertificate)
	# look at $tcpsocket.Client.RemoteEndPoint to see Server IP the client is using for connection
	$certinfo | fl
	$certinfo.Extensions | where {$_.Oid.FriendlyName -like 'subject alt*'} | `
	foreach { $_.Oid.FriendlyName; $_.Format($true) }
	$tcpsocket.Close() # Dispose() missing from older .NET, use Close()
	}
	main # run late-binding function defined at top of script
	# Based on https://blogs.technet.microsoft.com/parallel_universe_-_ms_tech_blog/2014/06/26/reading-a-certificate-off-a-remote-ssl-server-for-troubleshooting-with-powershell/
	# Subject Alternative Name info is displayed using technique suggested by Craig Landis
	# … https://social.technet.microsoft.com/wiki/contents/articles/1447.display-subject-alternative-names-of-a-certificate-with-powershell.aspx

view raw

Get-Cert.ps1

hosted with ❤ by GitHub

Here we use System.Net.Sockets.TcpClient with System.Net.Security.SslStream to open a secure session using the AuthenticateAsClient method. We then use System.Security.Cryptography.X509Certificates.X509Certificate2 to read more detail from the server certificate (RemoteCertificate property of SslStream). Getting “Subject Alternative Name” from Extensions property where we filter out the matching Oid FriendlyName – similar to example: Display Subject Alternative Names of a Certificate with PowerShell (Craig Landis, TechNet Wiki). Common ports to test include 443 (https) and 3389 (rdp).

For more detail and a discussion of potential error handling please see the original post linked above from Mr McCabe. Enjoy!

Verify Server Certificate with OpenSSL

Here’s a quick note for checking the contents of a TLS (SSL) server certificate with the OpenSSL command-line tools.

	echo "Q" | openssl s_client -connect "your.fqdn.com:443" \
	| openssl x509 -noout -text

view raw

verify-tls.sh

hosted with ❤ by GitHub

Comments: echo “Q” causes the openssl s_client to disconnect (quit) immediately after verifying connectivity. Replace your hostname and port for “your.fqdn.com:443”. The x509 command shows the contents of the certificate in text format.

This should return details for the server cert like: Issuer, Validity (Not Before, Not After), Subject (CN), Subject Alternative Name (SAN), etc. This is useful for verifying changes after installing a new cert or checking values for your existing certs (expiration, …).

Linux users will have openssl pre-installed. Windows users can find these tools in Git Bash, or with the latest Windows 10 in Windows Subsystem for Linux (WSL) distribution like Ubuntu.

Cisco ASA Troubleshoot IKE Policy

Notes for troubleshooting Cisco ASA IKE Policy – there must be a match between site-to-site / lan-to-lan (L2L) endpoint IPSec Peers for the ISAKMP (IKE) session to be established. Traditionally this was known of IPSec Phase One, but as of IKEv2 there may be new terminology to describe establishing an IKE Session between peers? This type of troubleshooting is especially important when attempting to connect to non-Cisco or 3rd-party endpoints like AWS, Azure, etc.

It appears that the Cisco ASA debug and log messages do not provide enough detail to identify specific IKE policies presented by peers when attempting to establish an ISAKMP session. In order to capture these proposed IKE policies, use the ASA capture ability. Substitute your own capture name in place of CAP_NAME.

• Define an access-list to identify interesting traffic by IP or protocol – usually bi-directional between the Cisco ASA outside IP and other IKE endpoint.
• Create a capture using the capture command and previously defined access-list on the ASA IKE interface (usually “outside”).
• Use the “show capture CAP_NAME” command to view a list of packets in the capture buffer. If needed, make adjustments to the access-list and use “clear capture CAP_NAME” command to remove the erroneous packets from the capture buffer.
• Use “capture CAP_NAME stop” and “no capture CAP_NAME stop” to stop and re-start the capture if needed.
• Use the “no capture CAP_NAME” command to remove the capture when no longer needed – you may need to stop and/or clear it to fully remove it. Use “show capture” command to verify that the capture is no longer present.
• Copy a pcap file to the ASA flash using “copy /pcap capture:CAP_NAME flash:/CAP_NAME.pcap” command.
• Use “scp” or similar to transfer your pcap file to a computer for review with Wireshark. The standard Packet Detail view for ISAKMP (IKE) in Wireshark will have the details of IKE policies proposed by both peers when attempting to establish an ISAKMP/IKE Session.

Once the ASA and peer endpoint agree upon a compatible IKE/ISAKMP policy, the phase one session will be established. This will be visible with the “show crypto ikev2 sa” command (assuming IKEv2). You can filter with “| begin 1.2.3.4” to look for a specific peer IP of 1.2.3.4 (substitute your peer IP).

Troubleshooting Phase Two IPSec ESP Data Sessions is a little easier with the ASA (ESP = Encapsulating Security Payload IPSec Security Associations). The standard warning and error logs are usually sufficient to determine what the problem might be (traffic list doesn’t match, proposal doesn’t match, etc). Use commands like “logging enable” “logging asdm warnings” and “show log asdm” commands to view log messages or manage log settings. There are other ways to interact with logs on the ASA, but the built-in asdm log buffer is one of the easiest to work with.

Good luck with your Cisco ASA IKE troubleshooting sessions!

Set Windows Firewall Zone to Domain

The Windows native host-based firewall is zone-based. This would be fine if you could set the zone for a given interface, but Microsoft has designed it to automatically determine the zone for a network interface using their Network Location Awareness (NLA) zone-detection feature. If a domain-joined computer cannot reach and authenticate to a domain controller when a network interface comes online, it will choose either public or private zone for the connection (public is default). Unfortunately the public zone is very restrictive and will likely block things like ping (icmp echo), remote desktop (tcp 3389 “rdp”), and file sharing (smb). Without these basic services available, you might need physical console access to a system to restore connectivity or update firewall rules.

Here are some steps to force a modern Windows system to re-identify a connection as a domain network (Server 2012+, Windows 8+, Windows 10). It is assumed that you are logged onto the system using some method either via a physical console, virtual console, or remotely managing if firewall allows.

• Set the connection-specific dns suffix in the “Advanced” network settings for the interface to MATCH the fully qualified DNS domain the computer is a member of.
  • The Get-DnsClient and Set-DnsClient cmdlets in PowerShell can be used to view and set the ConnectionSpecificSuffix from command-line.
• Open an elevated (run-as-administrator) PowerShell and use get-netadapter and restart-netadapter commands to find and disable-enable (cycle) the interface for re-identification by Windows.

Using these simple steps you should be able to help Windows correctly identify a Domain network interface – assuming a domain controller is available for authentication through that connection. If you have critical network services on your system and you know for a fact that all of the system interfaces will never be off of your domain network – you can add or update firewall rules on the “Advanced” tab to belong to “All” zones (Domain + Private + Public). If a rule is enabled for All zones, then even when Windows mis-identifiec a network interface – the traffic will still be allowed.

Thanks to Evan Barr for his blog article “Windows Server – Force Your Network Connection to Where it Belongs.” This post was inspired by his informative and helpful post on the topic. Good luck with your Windows Firewall tasking!

Windows Tomcat Manager GUI non-Admin

To workaround UAC limitations, the Apache Tomcat Monitor GUI app (tomcat8w/tomcatNw) has an embedded manifest to force elevation (requestedExecutionLevel = requireAdministrator). You can view the manifest settings in the Apache Commons Daemon Procrun prunmgr source code.

Because the manifest is embedded, it is a little tricky to run the executable WITHOUT elevating as an administrator. A basic way to run the program without elevation is to wrap it in a batch file that starts with

set __COMPAT_LAYER=RunAsInvoker
C:\path\to\tomcat8w.exe

Read a discussion of the workaround on superuser: Force a program to run without administrator privileges or UAC?

NOTE that running the Tomcat Monitor as non-admin will lack necessary permissions. You must separately grant the user access to manage the related Tomcat service, write to the related procrun registry location for the tomcat instance, and usually write the the Tomcat configuration and deployment directories. Without the necessary permissions, the designated user will not be able to fully manage the selected Tomcat service instance.

Registry paths that may require added user permissions include:

• HKLM:\SYSTEM\CurrentControlSet\services\YOUR-TOMCAT-INSTANCE
• HKLM:\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\YOUR-TOMCAT-INSTANCE

The non-registry service permissions can be managed with the sc.exe sdshow and sc.exe sdset commands. Specifically adding an entry for your SID to the SDDL output of sdshow right before the S: audit entries. Something like (A;;GRGX;;;S-1-2-34-…-…-…-…) which grants Allow of Generic-Read and Generic-Execute required by the Tomcat Monitor GUI. A discussion of how to use sc.exe and SDDL is beyond the scope of this blog post, sorry.

This reminds me that the security defaults for Tomcat on Windows are relatively weak. The default service account selected by the Tomcat Windows Installer is “Local System account” which is the highest privilege account on any Windows system. The Windows installer and GUI Manager tool are built with the expectation that SYSTEM will be the service account and that users managing the Tomcat instance will have Administrator-level privileges on the Windows system. It would be preferred to follow the principle of least-privilege and run Tomcat under an account that does not have broad access to the host system.

PowerShell Foreach-Object Modify Pipeline Objects

You can modify pipeline objects on-the-fly using Foreach-Object. The syntax is a little strange because you need to put the object back on the pipeline if desired for additional processing. WordPress.com code-sanitization is eating my $_ so trying Gist, BLAH.

	# This example changes name "frank" on pipeline objects to "francis"
	$objList | foreach { if($_.name -eq "frank") { $_.name = "francis" } $_ } | sort name

view raw powershell-foreach-pipeline.ps1 hosted with ❤ by GitHub

The key thing to remember here is that the extra $_ at the end of the foreach script block is used to put the modified object back on the pipeline. This is then available to the sort cmdlet and any later pipeline operations.

The pipeline concept in powershell can be a little strange at first, but it is a powerful way to work quickly with structured data.