Cisco ASA Troubleshoot IKE Policy

Notes for troubleshooting Cisco ASA IKE Policy – there must be a match between site-to-site / lan-to-lan (L2L) endpoint IPSec Peers for the ISAKMP (IKE) session to be established. Traditionally this was known of IPSec Phase One, but as of IKEv2 there may be new terminology to describe establishing an IKE Session between peers? This type of troubleshooting is especially important when attempting to connect to non-Cisco or 3rd-party endpoints like AWS, Azure, etc.

It appears that the Cisco ASA debug and log messages do not provide enough detail to identify specific IKE policies presented by peers when attempting to establish an ISAKMP session. In order to capture these proposed IKE policies, use the ASA capture ability. Substitute your own capture name in place of CAP_NAME.

  • Define an access-list to identify interesting traffic by IP or protocol – usually bi-directional between the Cisco ASA outside IP and other IKE endpoint.
  • Create a capture using the capture command and previously defined access-list on the ASA IKE interface (usually “outside”).
  • Use the “show capture CAP_NAME” command to view a list of packets in the capture buffer. If needed, make adjustments to the access-list and use “clear capture CAP_NAME” command to remove the erroneous packets from the capture buffer.
  • Use “capture CAP_NAME stop” and “no capture CAP_NAME stop” to stop and re-start the capture if needed.
  • Use the “no capture CAP_NAME” command to remove the capture when no longer needed – you may need to stop and/or clear it to fully remove it. Use “show capture” command to verify that the capture is no longer present.
  • Copy a pcap file to the ASA flash using “copy /pcap capture:CAP_NAME flash:/CAP_NAME.pcap” command.
  • Use “scp” or similar to transfer your pcap file to a computer for review with Wireshark. The standard Packet Detail view for ISAKMP (IKE) in Wireshark will have the details of IKE policies proposed by both peers when attempting to establish an ISAKMP/IKE Session.

Once the ASA and peer endpoint agree upon a compatible IKE/ISAKMP policy, the phase one session will be established. This will be visible with the “show crypto ikev2 sa” command (assuming IKEv2). You can filter with “| begin” to look for a specific peer IP of (substitute your peer IP).

Troubleshooting Phase Two IPSec ESP Data Sessions is a little easier with the ASA (ESP = Encapsulating Security Payload IPSec Security Associations). The standard warning and error logs are usually sufficient to determine what the problem might be (traffic list doesn’t match, proposal doesn’t match, etc). Use commands like “logging enable” “logging asdm warnings” and “show log asdm” commands to view log messages or manage log settings. There are other ways to interact with logs on the ASA, but the built-in asdm log buffer is one of the easiest to work with.

Good luck with your Cisco ASA IKE troubleshooting sessions!

Posted in Networking | Tagged , , | Leave a comment

Set Windows Firewall Zone to Domain

The Windows native host-based firewall is zone-based. This would be fine if you could set the zone for a given interface, but Microsoft has designed it to automatically determine the zone for a network interface using their Network Location Awareness (NLA) zone-detection feature. If a domain-joined computer cannot reach and authenticate to a domain controller when a network interface comes online, it will choose either public or private zone for the connection (public is default). Unfortunately the public zone is very restrictive and will likely block things like ping (icmp echo), remote desktop (tcp 3389 “rdp”), and file sharing (smb). Without these basic services available, you might need physical console access to a system to restore connectivity or update firewall rules.

Here are some steps to force a modern Windows system to re-identify a connection as a domain network (Server 2012+, Windows 8+, Windows 10). It is assumed that you are logged onto the system using some method either via a physical console, virtual console, or remotely managing if firewall allows.

  • Set the connection-specific dns suffix in the “Advanced” network settings for the interface to MATCH the fully qualified DNS domain the computer is a member of.
    • The Get-DnsClient and Set-DnsClient cmdlets in PowerShell can be used to view and set the ConnectionSpecificSuffix from command-line.
  • Open an elevated (run-as-administrator) PowerShell and use get-netadapter and restart-netadapter commands to find and disable-enable (cycle) the interface for re-identification by Windows.

Using these simple steps you should be able to help Windows correctly identify a Domain network interface – assuming a domain controller is available for authentication through that connection. If you have critical network services on your system and you know for a fact that all of the system interfaces will never be off of your domain network – you can add or update firewall rules on the “Advanced” tab to belong to “All” zones (Domain + Private + Public). If a rule is enabled for All zones, then even when Windows mis-identifiec a network interface – the traffic will still be allowed.

Thanks to Evan Barr for his blog article “Windows Server – Force Your Network Connection to Where it Belongs.” This post was inspired by his informative and helpful post on the topic. Good luck with your Windows Firewall tasking!

Posted in System Administration | Tagged , , | Leave a comment

Windows Tomcat Manager GUI non-Admin

To workaround UAC limitations, the Apache Tomcat Monitor GUI app (tomcat8w/tomcatNw) has an embedded manifest to force elevation (requestedExecutionLevel = requireAdministrator). You can view the manifest settings in the Apache Commons Daemon Procrun prunmgr source code.

Because the manifest is embedded, it is a little tricky to run the executable WITHOUT elevating as an administrator. A basic way to run the program without elevation is to wrap it in a batch file that starts with

set __COMPAT_LAYER=RunAsInvoker

Read a discussion of the workaround on superuser: Force a program to run without administrator privileges or UAC?

NOTE that running the Tomcat Monitor as non-admin will lack necessary permissions. You must separately grant the user access to manage the related Tomcat service, write to the related procrun registry location for the tomcat instance, and usually write the the Tomcat configuration and deployment directories. Without the necessary permissions, the designated user will not be able to fully manage the selected Tomcat service instance.

Registry paths that may require added user permissions include:

  • HKLM:\SYSTEM\CurrentControlSet\services\YOUR-TOMCAT-INSTANCE
  • HKLM:\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\YOUR-TOMCAT-INSTANCE

The non-registry service permissions can be managed with the sc.exe sdshow and sc.exe sdset commands. Specifically adding an entry for your SID to the SDDL output of sdshow right before the S: audit entries. Something like (A;;GRGX;;;S-1-2-34-…-…-…-…) which grants Allow of Generic-Read and Generic-Execute required by the Tomcat Monitor GUI. A discussion of how to use sc.exe and SDDL is beyond the scope of this blog post, sorry.

This reminds me that the security defaults for Tomcat on Windows are relatively weak. The default service account selected by the Tomcat Windows Installer is “Local System account” which is the highest privilege account on any Windows system. The Windows installer and GUI Manager tool are built with the expectation that SYSTEM will be the service account and that users managing the Tomcat instance will have Administrator-level privileges on the Windows system. It would be preferred to follow the principle of least-privilege and run Tomcat under an account that does not have broad access to the host system.

Posted in System Administration | Tagged , | Leave a comment

PowerShell Foreach-Object Modify Pipeline Objects

You can modify pipeline objects on-the-fly using Foreach-Object. The syntax is a little strange because you need to put the object back on the pipeline if desired for additional processing. code-sanitization is eating my $_ so trying Gist, BLAH.

The key thing to remember here is that the extra $_ at the end of the foreach script block is used to put the modified object back on the pipeline. This is then available to the sort cmdlet and any later pipeline operations.

The pipeline concept in powershell can be a little strange at first, but it is a powerful way to work quickly with structured data.

Posted in System Administration | Tagged | Leave a comment

JIRA Export Import Project Issues

This is a topic that seems to be poorly documented – exporting and importing projects between JIRA instances. There is an entire system backup, but the documentation makes it clear that restoring individual projects from the system backup is not fully supported. The best method our team has come up with is to use the CSV export capability of the JIRA issue search functionality – this export is designed to match the CSV “external system” import for JIRA projects. Both issues and associated comments can be exported in this way. The default field names for both the export and import are likely to be acceptable for most users. The project must be created on target JIRA prior to issue and comment import. If you have trouble importing, you may need to configure the new target project to accommodate any special issue fields or project settings required for your issues and comments to import. Here is some related Atlassian documentation:

Good luck importing and exporting project issues with JIRA. It’s strange to me that native JIRA issue import/export or project import/export are not supported by Atlassian other than this generic CSV capability. It may be a little rough, but the process works reliably and is similar for importing or exporting data from other issue tracking systems missing from the external system import list in JIRA.

Posted in System Administration | Tagged , | Leave a comment

Exchange Offline Address Book Troubleshooting

The Exchange Offline Address Book (OAB) for Outlook clients can be a difficult beast to troubleshoot. When it’s not working correctly, Outlook clients may fail to report an error and continue to synchronize stale Global Address List records. Here are some hints for successful resolution of OAB issues – tested with Exchange 2010.

  • Try following the Corelan document: Fixing Exchange 2007 Offline Address Book generation (oalgen) and distribution issues
    • This applies equally well to Exchange 2010 although some file paths may have changed slightly.
  • Before intensive troubleshooting, try restarting the “MSExchangeSA” System Attendant service on the server responsible for generating the Offline Address Book. If Windows Updates are pending reboot, just reboot Windows instead.
  • Following a clean restart of the System Attendant on your OAB Generation Server, use the Exchange Management Console (EMC) to manually start an “Update” of the OAB under Organization Configuration – Mailbox – Offline Address Book.
    • Status of the Offline Address Book generation should show in the Windows “Application” Event Log with Source “MSExchangeSA” – wait for indication that the OAB updates have completed.
  • If the OAB generation completes successfully, you can force synchronization to each CAS server by Restarting the “MSExchangeFDS” File Distribution Service on each CAS server. Review the Windows “Application” event log for messages from Source “MSExchangeFDS” that will indicate whether the new OAB has finished copying.
    • Default “polling” interval for FDS is 8 hours (480 minutes) – this is a long time to wait for testing!
  • Review the OAB file dates. Many should have been updated within the last 24 hours. Your system may use slightly different paths.
    • On the OAB Generation Server under C:\Program Files\Microsoft\Exchange Server\V14\ExchangeOAB\*\*.* (these are created by the System Attendant)
    • On the CAS servers that distribute your address book under C:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\OAB\*\*.* (these are created by the File Distribution Service)

There is a lot more to this troubleshooting than I’m mentioning here. I highly recommend you read and follow the above linked Corelan document – it was very helpful to me for resolving this issue.

Posted in System Administration | Tagged , | Leave a comment

Cisco ASA Command Line Basics

This post is for people who are new to the Cisco ASA command line, or seasoned network administrators like myself who need to review or brush up on the command line basics for the ASA console. Instead of using my own words, I’ll just refer to the official documentation as a good reference and intro.

This is a great resource because it covers many common tricks and techniques to make the best use of your ASA device. CLI skills obtained from this article help us quickly accomplish complex tasks rather than stumbling through basic device configuration.

Posted in Networking, System Administration | Tagged , , | Leave a comment

AES-GCM on Cisco ASA

This is a request for comments to clarify network security proper usage of new AES-GCM cryptography functionality on the Cisco ASA platform. Please leave a comment if you can provide some insight to help readers better informed on how and when to use AES-GCM with the Cisco ASA. I’m using the documentation for reference: “CLI Book 3: Cisco ASA Series VPN CLI Configuration Guide, 9.7 – Chapter: IPsec and ISAKMP.” Quotes that I’m trying to decipher from the document follow:

  • “When AES-GCM is specified as the encryption algorithm, an administrator can choose null as the IKEv2 integrity algorithm” – under section “IKE Policy Keywords and Values”
  • “You must choose the null integriy algorithm if AES-GCM/GMAC is configured as the encryption algorithm” – under section “Create Static Crypto Maps”
  • “You must choose the null integrity algorithm if AES-GCM/GMAC has been configured as the encryption algorithm” – under section “Create Static Crypto Maps”

For the ikev2 policy, this is fairly simple because the only allowable integrity method is “null” when encryption type is set to aes-gcm-*. Because the software enforces this requirement, there is no need to clarify the ambiguity in the documentation.

For the ipsec ipsec-proposal, this still remains confusing to me. When the esp encryption is set to aes-gcm-*, the ASA software allows configuration of integrity types other than “null” … this seems to be in violation of the documentation which states “YOU MUST CHOOSE THE NULL INTEGRITY ALGORITHM [when encrypting with aes-gcm-*].” There is an example further down in the documentation that seems to suggest the correct usage of aes-gcm proposal with null integrity algorithm specified.

ciscoasa(config)# show run crypto ipsec 
 crypto ipsec ikev2 ipsec-proposal GCM 
 protocol esp encryption aes-gcm
 protocol esp integrity null

I think I may need to fall back on a more basic question like: “What is the AES-GCM cipher is and why would I want to use it?” According to a Wikipedia article, the Galois/Counter Mode (GCM) is “adopted because of its efficiency and performance and is “designed to provide both data authenticity (integrity) and confidentiality. This would explain why we use “null” integrity under ESP configuration – the integrity is already built into the encryption algorithm. Any integrity method specified for a GCM ESP session would be adding overhead because GCM is already providing message authentication for the encrypted channel. Because the ASA allows an integrity method to be specified here, I think this is allowing users to mis-configure the device ESP and establish tunnels with performance-reducing double integrity checks.

Update for ASA 7.1 software. When configuring aes-gcm* for esp, there is an informative message regarding esp integrity selection: “WARNING: GCM\GMAC are authenticated encryption algorithms. esp integrity config is ignored.” This helps explain that if the peers negotiate gcm for the security association, then none of the configured integrity methods will be used.

Regarding why we want to use it, the same article states: “GCM can take full advantage of parallel processing.” This is the real reason we want to use GCM – it’s FASTER THAN TRADITIONAL AES ON MULTI-CORE NETWORK DEVICES! There appears to be no real security advantage of AES-GCM over AES-CBC. The original AES Cipher Block Chain was designed as a sequential algorithm best suited to execution in a single process (single-core hardware utilization). This distinction makes traditional AES a good choice on single-core devices. AES-GCM should be considered for performance and throughput improvements on multi-core network hardware. Even on systems with less cores, GCM may be a good choice if a multi-core or pipelined cryptography accelerator module is used to provide high-performance gcm cipher features for your IPSec traffic.

A side note, the Wikipedia article also explains that GMAC is “an authentication-only variant of the GCM.” In my opinion, Cisco should make it clear to customers that GMAC DOES NOT ENCRYPT YOUR DATA!!! Because of this, I recommend that Cisco ASA users avoid the use of the GMAC non-cipher. Good news is that gmac is not available under your ikev2 policy. Unfortunately the gmac is a selection under your ipsec-proposal, so please avoid it if you want your data to remain private!!

Posted in Networking | Tagged , , , | Leave a comment

SSSD-AD TGT failed verification

Users of RHEL 7 and CentOS 7 on Windows Active Directory networks are likely enjoying the benefits of using the SSSD-AD domain-join client module along with the Realmd tool which facilitates proper management of SSSD client configuration (a very complex task).

Unfortunately, with Enterprise domain services like Active Directory (AD), there are MANY things that can go wrong (Murphy’s Law). Every AD domain is unique because the AD is highly customizable. One such thing that can go wrong is Kerberos Ticket Validation. TGT verification failure is such a common issue that Red Hat has dedicated a (customers-only) solution page to it: SSSD user logins fail due to failed TGT validation.

As of March 2017, their suggestions failed to mention a solution that I needed on one of my systems. I’m recording the issue I saw here so I can refer back to it.

  • Set debug_level to a high value like 6 for all sections in your sssd.conf file
  • Restart sssd “systemctl restart sssd”
  • Attempt to login to your system with a domain user account
  • Review the contents of /var/log/sssd/krb5_child for indicators of this problem
    • validate_tgt” line like “TGT failed verification using key for [host/your.fqdn@YOUR.REALM]
    • Additional lines like “Server not found in Kerberos database
  • I’m not sure if you can get more useful log output by setting the debug level higher – this was the most detail I found before solving the problem
    • Duplicate Service Principal Names will be indicated in the output of Windows command “setspn -X” when run by a domain admin user.
    • If duplicate SPN’s are found, they must be resolved
      • EITHER delete unused duplicate objects from AD
      • OR leave domain (realm leave) – change hostname (nmtui) – reboot – rejoin domain (realm join) with Linux client
  • This security validation of the Ticket Granting Ticket (TGT) is controlled by the setting “krb5_validate” (true or false) in the domain-specific section of your “sssd.conf” file. You can change this setting and restart sssd to test whether the TGT validation checks are causing your issue. There are many factors that may cause this validation to fail – Duplicate SPN is one such issue. For other possibilities, please review the above linked Red Hat solution, or try your luck with Google :-).
  • Once the underlying issue is resolved, set krb_validate back to true

I hope this is helpful to someone experiencing the same issue. As we can see here, a cleanup of the Active Directory infrastructure resolves the problem (removal of duplicate SPN in this case). Other basic infrastructure services should all be in place and functioning correctly to include: Forward and Reverse DNS Name Resolution, Secure Dynamic DNS Name Registration, Domain Time Synchronization (PDC authoritative, automatic to Windows domain members, NTP/chronyd to Linux clients)

Posted in Linux, System Administration | Tagged , , , , | Leave a comment

Get Rid of virbr0

In RHEL 7.x and CentOS 7.x you may see an odd extra network interface listed as “virbr0” (virtual bridge zero). This is provided as a default way to share the host physical network with private guest virtual machines. Unfortunately this interface is default assigned an IP and shows up as an active interface even when you’re not running any virtual machines.

If you’re not using the RHEL/CentOS virtualization feature, I recommend you turn off libvirtd which will get rid of this odd extra interface. This may be particularly useful to Enterprise Domain users who would like to prevent this Non-Routable IP from being registered in the organization’s DNS Infrastructure causing reachability and name resolution trouble for network users.

  • systemctl disable libvirtd
  • systemctl stop libvirtd
  • shutdown -r now

After a reboot, the virbr0 will be gone from your system and the network configuration will be clean again. I’m not sure if there is any easy way to reload the network without rebooting – feel free to comment if you have a reliable supported way to finish this task without rebooting the system.

If you’re making use of the virtualization functionality, you can still restrict which interfaces are used for Dynamic DNS registration. Use the “dyndns_iface” option in your “sssd.conf”.

Posted in Linux, Networking, System Administration | Tagged , , , | Leave a comment