On Juniper Networks PTX and QFX Series devices with packet sampling configured using tunnel-observation mpls-over-udp, sampling of a malformed packet can cause the Kernel Routing Table (KRT) queue to become stuck. KRT is the module within the Routing Process Daemon (RPD) that synchronized the routing tables with the forwarding tables in the kernel. This table is then synchronized to the Packet Forwarding Engine (PFE) via the KRT queue. Thus, when KRT queue become stuck, it can lead to unexpected packet forwarding issues. An administrator can monitor the following command to check if there is the KRT queue is stuck: user@device > show krt state ... Number of async queue entries: 65007 <--- this value keep on increasing. When this issue occurs, the following message might appear in the /var/log/messages: DATE DEVICE kernel: %KERN-3: rt_pfe_veto: Too many delayed route/nexthop unrefs. Op 2 err 55, rtsm_id 5:-1, msg type 2 DATE DEVICE kernel: %KERN-3: rt_pfe_veto: Memory usage of M_RTNEXTHOP type = (0) Max size possible for M_RTNEXTHOP type = (7297134592) Current delayed unref = (60000), Current unique delayed unref = (18420), Max delayed unref on this platform = (40000) Current delayed weight unref = (60000) Max delayed weight unref on this platform= (400000) curproc = rpd This issue affects Juniper Networks Junos OS on PTX/QFX Series: 17.2X75 versions prior to 17.2X75-D105; 18.1 versions prior to 18.1R3-S11; 18.2 versions prior to 18.2R3-S5; 18.2X75 versions prior to 18.2X75-D420, 18.2X75-D53, 18.2X75-D65; 18.3 versions prior to 18.3R2-S4, 18.3R3-S3; 18.4 versions prior to 18.4R1-S7, 18.4R2-S5, 18.4R3-S4; 19.1 versions prior to 19.1R2-S2, 19.1R3-S2; 19.2 versions prior to 19.2R1-S5, 19.2R3; 19.3 versions prior to 19.3R2-S3, 19.3R3; 19.4 versions prior to 19.4R1-S2, 19.4R2-S1, 19.4R3; 20.1 versions prior to 20.1R1-S2, 20.1R2. This issue does not affect Juniper Networks Junos OS prior to 18.1R1.
With Rapid7 live dashboards, I have a clear view of all the assets on my network, which ones can be exploited, and what I need to do in order to reduce the risk in my environment in real-time. No other tool gives us that kind of value and insight.
– Scott Cheney, Manager of Information Security, Sierra View Medical Center