Citrix NetScaler traffic capture using nstrace and nstcpdump

Nstrace

Nstrace is a NetScaler packet capture tool. Nstrace dumps packets in the native NetScaler format. These trace files have an extension of .cap and can be analysed with WireShark. You can use specific filters in WireShark as normal to filter through captured data or specify filters using the NetScaler CLI. This allows you to only capture traffic of interest.

start nstrace – Captures all traffic.

stop nstrace – Stops the packet capture.

show nstrace – Shows the status of nstrace and if it is running etc.

1-min

Some classic nstrace expressions:

start nstrace -filter “vsvrname == LDAP-LoadBalanced-vServer” – Captures traffic to and from the specified vServer.

2-min

3-min

start nstrace -size 0 -filter “svcname == xx” – Captures traffic to and from the specified service. -size 0 means all packets are captured regardless of packet size.

start nstrace -filter “DESTIP == 192.168.0.242” -link ENABLE – Captures all traffic to destination IP 192.168.0.242. The -link ENABLE switch is used to capture return traffic from 192.168.0.242.

4-min

5-min

start nstrace -filter “DESTIP == x.x.x.x” – Captures all traffic to destination IP.

start nstrace -filter “SRCIP == x.x.x.x” – Captures traffic sent from source IP.

start nstrace -filter “DESTPORT == 443” – Captures traffic where the destination port is 443.

6-min

7-min

start nstrace -filter “SRCIP == x.x.x.x && DESTPORT == xx” – Captures traffic from a specified source IP and a specified destination port.

Some default commands/expressions:

Note: These expressions are the same onces you can use to filter traffic when running nstrace from the GUI. I prefer to use these expressions.

start nstrace -filter CONNECTION.SRCIP.EQ(192.168.0.100) -link ENABLED – Captures all traffic to and from 192.168.0.100.

start nstrace -filter CONNECTION.SRCIP.NE(192.168.0.100) – Captures all source traffic apart from when the source is 192.168.0.100.

start nstrace -filter “CONNECTION.SRCIP.EQ(192.168.0.100) || CONNECTION.DSTPORT.EQ(443)” – Captures all traffic from 192.168.0.100 with the destination port as 443.

start nstrace -filter “CONNECTION.DSTIP.EQ(192.168.0.15) || CONNECTION.DSTIP.EQ(192.168.0.16) || CONNECTION.DSTIP.EQ(192.168.0.17)” -link ENABLED – Captures traffic to and from three IP addresses.

start nstrace -filter CONNECTION.INTF.EQ(“0/1”) – Captures all traffic flowing in and out of interface 0/1.

start nstrace -filter CONNECTION.VLANID.EQ.(205) – Captures all traffic flowing in and out of VLAN 205.

start nstrace -filter CONNECTION.DSTPORT.BETWEEN(80,180) – Captures traffic when the destination port is between 80 and 180.

start nstrace -filter CONNECTION.LB_VSERVER.NAME.EQ(“name”) – Captures traffic to the specified Load Balanced Virtual Server.

start nstrace -filter CONNECTION.SERVICE_TYPE.EQ(SSL) – Captures all SSL traffic.

start nstrace -filter CONNECTION.SERVICE_TYPE.EQ(MONITOR_PING) – Captures all ping requests sent by a monitor including replies.

start nstrace -size 0 -nf 10 -time 120 – Starts an nstrace using circular logging. A maximum of 10 separate logs will be generates, each new log is generated every 120 seconds.

Nstcpdump

Nstcpdump can be used for more low-level troubleshooting. Nstcpdump does not collect as much detailed information as nstrace. Open NetScaler CLI and type shell. You can use filters with nstcpdump but cannot use filters specific to NetScaler resources. The dump output can be viewed directly within the CLI screen.

CTRL + C – Press these keys simultaneously to stop an nstcpdump.

nstcpdump.sh dst host x.x.x.x  – Shows traffic sent to the destination host .

nstcpdump.sh -n src host x.x.x.x – Shows traffic from specified host and don’t convert IP addresses to names (-n).

nstcpdump.sh host x.x.x.x – Shows traffic to and from specified host IP.

nstcpdump.sh -c 10 dst host 192.168.0.242 – Outputs the first 10 packets from destination 192.168.0.242.

8-min

nstcpdump.sh dst host 192.168.0.242 and port 443 – Outputs traffic destined for 192.168.0.242 on port 443.

9-min

nstcpdump.sh src host x.x.x.x and port xx – Outputs traffic from specified source IP and specified port .

nstcpdump.sh -e host x.x.x.x and host x.x.x.x – Shows traffic between two specified host IPs, and prints the MAC address (-e) of the interface sending traffic..

nstcpdump.sh icmp – Outputs all sent and replied echo requests. NetScaler uses ICMP to track up/down status of DNS servers configured on NetScaler for example.

Note: Pings to DNS come from the SNIP, using static routes to force these pings through the NSIP for example will not work.

10-min

nstcpdump.sh icmp and dst host x.x.x.x – Outputs all sent echo requests to a specified destination host.

nstcpdump.sh icmp and dst host x.x.x.x -w /var/nstrace/output.cap – Outputs all sent echo request to a specified destination host and saves the output in the specified cap file.

nstcpdump.sh udp and not port 3003 and not port 1985 – Outputs all UDP traffic but excludes traffic on port 3003 and 1985.

nstcpdump.sh port 53 | grep 192.168.1.1 – Outputs all traffic using port 53 from IP 192.168.1.1.


9 Comments

  • Andreas Fischer

    January 4, 2017

    Hi George! This page is very useful!
    I have a question regarding nstrace: why the SSLPLAIN-mode in NetScaler 11.1 is hidden? I read the Citrix documentation, unfortunately version 11.0, where it was described, but I didn’t found it on my 11.1-machine (VPX on an SDX 11500), neither in manual nor in command short help:
    NSVPX1_Primary#> start nstrace -nf 21 -time 120 -size 0 -mode
    APPFW
    C2C
    IPV6
    MPTCP
    NEW_RX
    NS_FR_TX
    RX
    TX
    TXB
    NSVPX1_Primary#>
    So I tried it nevertheless – and it’s functioning!! What does that mean? Will Citrix remove the SSLPLAIN-option in future or is it part of an other option?

    Thanks in advance for an answer and greetings from Dresden!
    Andreas

    Reply
  • George Spiers

    January 5, 2017

    Hi Andreas
    Yes, as you noticed SSLPLAIN still works as a command line switch. It is still also available from the GUI under System -> Diagnostics -> Start new trace, and it has been removed from 11.1 documentation.
    I don’t know for sure if it will be removed in the future however there is a different (newer) method to decrypt SSL. A new switch was added to late NetScaler v11.0.66.x and above versions. It is called -capsslkeys. So a command such as start nstrace -capsslkeys ENABLED would capture a trace file, and a second file named nstrace.sslkeys. You can import the session keys file in to WireShark to decrypt the captured file.

    Reply
  • Brandon

    January 12, 2018

    NetScaler – VIPS/GW Monitors going UP/DOWN

    Hi George,

    At times at least once or twice a week we see our vips/gslb/monitors flaps from one location (it varies on the site). But any recommendation how to troubleshoot this? We lose our vdi’s when it flaps and sometimes we don’t even notice. But if you notice below it shows Packets dropped in Rx (sw). Anything that stands out? Once it comes back it’s fine and can’t reproduce this issue at all.

    > stat interface 1/1

    Interface [1/1]:
    Interface State UP
    Link uptime 6.14:14:43
    Link downtime 00:00:11

    Throughput Statistics
    Rate (/s) Total
    Bytes received 435 1966651435
    Bytes transmitted 0 21144
    Packets received 5 18391487
    Packets transmitted 0 368
    Jumbo Packets Received 0 0
    Jumbo Packets Transmitted 0 0
    Tagged Packets Received on Trunk 0 0
    Tagged Packets transmitted on Tr 0 0

    Packet Statistics
    Rate (/s) Total
    Multicast packets 1 861758
    NetScaler packets 4 17113835

    LACP Statistics
    Rate (/s) Total
    LACPDUs received 0 0
    LACPDUs transmitted 0 0

    Error Statistics
    Rate (/s) Total
    Error packets received (hw) 0 0
    Error packets transmitted (hw) 0 0
    Inbound packets discarded(hw) 0 0
    Outbound packets discarded(hw) 0 0
    Packets dropped in Rx (sw) 1 658685
    Packets dropped in Tx (sw) 0 0
    NIC hangs — 0
    Status stalls — 0
    Transmit stalls — 0
    Receive stalls — 0
    Error-disables — 0
    Duplex mismatches — 0
    Link re-initializations — 0
    MAC moves registered 0 0
    Times NIC became muted — 0
    Receive CRC errors 0 0
    Done
    > stat interface 0/1

    Interface [0/1]:
    Interface State UP
    Link uptime 6.14:15:25
    Link downtime 00:00:11

    Throughput Statistics
    Rate (/s) Total
    Bytes received 689763 469703365313
    Bytes transmitted 30212 21854551192
    Packets received 510 349807958
    Packets transmitted 330 251322623
    Jumbo Packets Received 0 0
    Jumbo Packets Transmitted 0 0
    Tagged Packets Received on Trunk 0 0
    Tagged Packets transmitted on Tr 0 0

    Packet Statistics
    Rate (/s) Total
    Multicast packets 1 850560
    NetScaler packets 509 348938342

    LACP Statistics
    Rate (/s) Total
    LACPDUs received 0 0
    LACPDUs transmitted 0 0

    Error Statistics
    Rate (/s) Total
    Error packets received (hw) 0 0
    Error packets transmitted (hw) 0 0
    Inbound packets discarded(hw) 0 0
    Outbound packets discarded(hw) 0 0
    Packets dropped in Rx (sw) 0 299686
    Packets dropped in Tx (sw) 0 0
    NIC hangs — 0
    Status stalls — 0
    Transmit stalls — 0
    Receive stalls — 0
    Error-disables — 0
    Duplex mismatches — 0
    Link re-initializations — 0
    MAC moves registered 0 0
    Times NIC became muted — 0
    Receive CRC errors 0 0
    Done
    > stat ns | grep Megabits
    Megabits received 7 3847569
    Megabits transmitted 0 223267
    > stat ns | grep Megabits
    Megabits received 8 3847622
    Megabits transmitted 0 223270
    > stat ns | grep Megabits
    Megabits received 8 3847622
    Megabits transmitted 0 223270
    > stat ns | grep Megabits
    Megabits received 8 3847622
    Megabits transmitted 0 223270
    > stat ns | grep Megabits
    Megabits received 8 3847622
    Megabits transmitted 0 223270
    > stat ns | grep Megabits
    Megabits received 8 3847622
    Megabits transmitted 0 223270
    > stat ns | grep Megabits
    Megabits received 8 3847622
    Megabits transmitted 0 223270
    > stat ns | grep Megabits
    Megabits received 5 3847656
    Megabits transmitted 0 223272
    > stat ns | grep Megabits
    Megabits received 5 3847656
    Megabits transmitted 0 223272
    > stat ns | grep Megabits
    Megabits received 9 3847716
    Megabits transmitted 0 223274
    > stat ns | grep Megabits
    Megabits received 5 3848922
    Megabits transmitted 0 223333
    > stat ns | grep Megabits
    Megabits received 5 3848922
    Megabits transmitted 0 223333
    > stat interface 0/1

    Interface [0/1]:
    Interface State UP
    Link uptime 6.14:28:36
    Link downtime 00:00:11

    Throughput Statistics
    Rate (/s) Total
    Bytes received 1108259 470353531998
    Bytes transmitted 36337 21880328394
    Packets received 797 350283270
    Packets transmitted 486 251628450
    Jumbo Packets Received 0 0
    Jumbo Packets Transmitted 0 0
    Tagged Packets Received on Trunk 0 0
    Tagged Packets transmitted on Tr 0 0

    Packet Statistics
    Rate (/s) Total
    Multicast packets 1 851741
    NetScaler packets 796 349412446

    LACP Statistics
    Rate (/s) Total
    LACPDUs received 0 0
    LACPDUs transmitted 0 0

    Error Statistics
    Rate (/s) Total
    Error packets received (hw) 0 0
    Error packets transmitted (hw) 0 0
    Inbound packets discarded(hw) 0 0
    Outbound packets discarded(hw) 0 0
    Packets dropped in Rx (sw) 0 300103
    Packets dropped in Tx (sw) 0 0
    NIC hangs — 0
    Status stalls — 0
    Transmit stalls — 0
    Receive stalls — 0
    Error-disables — 0
    Duplex mismatches — 0
    Link re-initializations — 0
    MAC moves registered 0 0
    Times NIC became muted — 0
    Receive CRC errors 0 0
    Done
    > > show interface
    ^^^^^^^^^^^
    ERROR: No such command
    > shell
    Copyright (c) 1992-2013 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.

    root@# show interface

    > show interfaces

    1) Interface 0/1 (NetScaler Virtual Interface, VMXNET3) #0
    flags=0xc060
    MTU=1500, native vlan=1, MAC=00:50:56:be:40:c5, uptime 158h41m54s
    LLDP Mode: NONE, LR Priority: 1024

    RX: Pkts(350757538) Bytes(471004503134) Errs(0) Drops(300522) Stalls(0)
    TX: Pkts(251929029) Bytes(21905417051) Errs(0) Drops(0) Stalls(0)
    NIC: InDisc(0) OutDisc(0) Fctls(0) Stalls(0) Hangs(0) Muted(0)
    Bandwidth thresholds are not set.

    2) Interface 1/1 (NetScaler Virtual Interface, VMXNET3) #1
    flags=0xc060
    MTU=1500, native vlan=249, MAC=00:50:56:be:a6:6d, uptime 158h41m54s
    LLDP Mode: NONE, LR Priority: 1024

    RX: Pkts(18403708) Bytes(1967730206) Errs(0) Drops(660641) Stalls(0)
    TX: Pkts(369) Bytes(21204) Errs(0) Drops(0) Stalls(0)
    NIC: InDisc(0) OutDisc(0) Fctls(0) Stalls(0) Hangs(0) Muted(0)
    Bandwidth thresholds are not set.

    3) Interface LO/1 (Netscaler Loopback interface) #2
    flags=0x20008020
    MTU=1500, native vlan=1, MAC=00:50:56:be:40:c5, uptime 158h42m05s
    LLDP Mode: NONE, LR Priority: 1024

    RX: Pkts(53362344) Bytes(8051270186) Errs(0) Drops(0) Stalls(0)
    TX: Pkts(434220711) Bytes(476838601061) Errs(0) Drops(0) Stalls(0)
    Bandwidth thresholds are not set.

    Done

    > show ver
    NetScaler NS12.0: Build 56.20.nc, Date: Dec 19 2017, 06:00:47
    Done
    >

    > show ver
    NetScaler NS12.0: Build 56.20.nc, Date: Dec 19 2017, 06:00:47

    Reply
    • George Spiers

      January 16, 2018

      What VPX license are you running and how many vCPUs have the VPX(s) been assigned? Is there any consistency to the drops for example they happen at a certain time of the week and do all VIPs drop? Are you running HA and if so make sure failovers are not happening at this time? What is vLAN 249 for, is that where all VIP and SNIPs reside? You have no stalls, mutes, hangs and no sign of NetScaler crashing and packets dropped in Rx does not look abnormal as this can indicate NetScaler dropping packets not destined for any of its owned MAC addresses, if operating in L3 mode.

      Reply
  • Brandon

    January 17, 2018

    George,

    We are running Enterprise license 25 MB =VPX
    Not running HA
    2 two sites and gslb enabled
    Running 2gig’s of RAM and 2 CPU (1 core per socket)
    249 is where all the vips live and vdi subnet live as well
    snip is on 247

    should i upgrade the VM’s memory and cpu?

    Reply
    • Brandon

      January 17, 2018

      Hardware version 7 (ESXI)

      Reply
    • George Spiers

      January 18, 2018

      Should be OK unless you are experiencing high traffic at certain times. You could monitor the appliances with NetScaler MAS which can send you alerts if something is wrong such as high CPU during the time when you are experiencing packet loss. Also, if there is a trend to when this happens set up an nstrace at the time to see if you can find what is happening to the packets, or log on to the GUI of NetScaler and look at CPU/Memory consumption. It is also worth checking other appliances at this time such as firewalls in between NetScaler.

      You should also generate a tech support file by running “show techsupport” from the CLI and uploading that to cis.citrix.com

      Reply
  • Petr Hrubý

    July 20, 2018

    It looks like the nstcpdump.sh runs /netscaler/nsapimgr which captures and saves all (!) packets to /var/nstrace/nstcpdump directory. This temporary file is then used as a source file for standard tcpdump which produces target output. Therefore even if you specify complex capturing filter, the filesystem is full very quickly …

    nstcpdump.sh -w scaneng1.cap dst host 10.5.50.76 and ‘tcp[tcpflags] & (tcp-syn) != 0’
    reading from file -, link-type EN10MB (Ethernet)

    /var: write failed, filesystem is full

    Reply
  • Petr Hrubý

    July 20, 2018

    Btw I think that -size 0 means that the whole packet size is captured. If you specify a number, the packet is truncated to the specific size specified by the number. But all packets are always captured.

    Reply

Leave a Reply to Brandon Cancel reply