在U200防火墙上,通过<U200>debug IP packet 查看防火墙将数据包已经发出,<U200>debug ipsec packet 提示已经进入隧道,但是没有收到SR6604路由器的回应报文。在U200上PING 对端192.168.0.207 测试时发现,出方向统计的ACL3000 命中数增长,入方向统计ACL 3010 的命中数不变,这也说明进入了隧道。
从目前的分析情况看,问题可能出在了中间链路或者SR6604路由器设备上。
由于SR6604路由器上打开<SR6604>debug ipsec packet 后cpu100%,长时间收集信息,担心会影响业务,而且信息量非常大,不便于分析,所以只开了一分钟的debug,但是没有抓到相关有用的信息。
通过下面的信息分析,从U200防火墙上<U200>PING -c 10 -a 10.26.108.1 192.168.0.207
<SR6604>display ipsec statistics
the security packet statistics:
input/output security packets: 25172172445/31796989123
input/output security bytes: 3997629875568/5772012932920
input/output dropped security packets: 7519790/655691(PING包期间,数值没有增加)
dropped security packet detail:
not enough memory: 228
can't find SA: 3252553
queue is full: 0
authentication has failed: 159064
wrong length: 0
replay packet: 483429
packet too long: 639371
wrong SA: 0
decrypt/encrypt failed: 0
ACL check failure: 3640836
<U200>PING -c 10 -a 10.26.108.1 192.168.0.207
PING 192.168.0.207: 56 data bytes, press CTRL_C to break(全部都丢了)
Request time out
Request time out
Request time out
Request time out
Request time out
Request time out
Request time out
Request time out
Request time out
Request time out
<SR6604>
*Jan 14 12:24:14:953 2016 SR6604-1 DPIPFWD/7/debug_case: -Slot=3;
Receiving, interface = GigabitEthernet3/1/7, version = 4, headlen = 20, tos = 0,
pktlen = 84, pktid = 41917, offset = 0, ttl = 255, protocol = 1,
checksum = 57432, s = 10.26.108.1, d = 192.168.0.207
prompt: Receiving IP packet
*Jan 14 12:24:14:954 2016 SR6604-1 DPIPFWD/7/debug_case: -Slot=3;
Delivering, interface = GigabitEthernet3/1/7, version = 4, headlen = 20, tos = 0,
pktlen = 84, pktid = 41917, offset = 0, ttl = 255, protocol = 1,
checksum = 57432, s = 10.26.108.1, d = 192.168.0.207
prompt: IP packet is delivering up!
*Jan 14 12:24:14:955 2016 SR6604-1 ADDR/7/debug_case: -Slot=3;
Delivering, interface = GigabitEthernet3/1/7, version = 4, headlen = 20, tos = 0,
pktlen = 84, pktid = 41917, offset = 0, ttl = 255, protocol = 1,
checksum = 57432, s = 10.26.108.1, d = 192.168.0.207
prompt: IP packet is delivering up!
*Jan 14 12:24:14:955 2016 SR6604-1 DPIPFWD/7/debug_case: -Slot=3;
Sending, interface = GigabitEthernet3/1/7, version = 4, headlen = 20, tos = 0,
pktlen = 84, pktid = 32330, offset = 0, ttl = 255, protocol = 1,
checksum = 57432, s = 192.168.0.207, d = 10.26.108.1
prompt: Sending the packet from local
*Jan 14 12:24:17:155 2016 SR6604-1 DPIPFWD/7/debug_case: -Slot=3;
Receiving, interface = GigabitEthernet3/1/7, version = 4, headlen = 20, tos = 0,
pktlen = 84, pktid = 41929, offset = 0, ttl = 255, protocol = 1,
checksum = 57420, s = 10.26.108.1, d = 192.168.0.207
prompt: Receiving IP packet
*Jan 14 12:24:17:155 2016 SR6604-1 DPIPFWD/7/debug_case: -Slot=3;
Delivering, interface = GigabitEthernet3/1/7, version = 4, headlen = 20, tos = 0,
pktlen = 84, pktid = 41929, offset = 0, ttl = 255, protocol = 1,
checksum = 57420, s = 10.26.108.1, d = 192.168.0.207
prompt: IP packet is delivering up!
*Jan 14 12:24:17:156 2016 SR6604-1 ADDR/7/debug_case: -Slot=3;
Delivering, interface = GigabitEthernet3/1/7, version = 4, headlen = 20, tos = 0,
pktlen = 84, pktid = 41929, offset = 0, ttl = 255, protocol = 1,
checksum = 57420, s = 10.26.108.1, d = 192.168.0.207
prompt: IP packet is delivering up!
*Jan 14 12:24:17:156 2016 SR6604-1 DPIPFWD/7/debug_case: -Slot=3;
Sending, interface = GigabitEthernet3/1/7, version = 4, headlen = 20, tos = 0,
pktlen = 84, pktid = 32392, offset = 0, ttl = 255, protocol = 1,
checksum = 57420, s = 192.168.0.207, d = 10.26.108.1
prompt: Sending the packet from local
从IP层面看SR6604路由器都回复了,现在重点排查SR6604路由器IPSec模块处理是否正常。
排查过程如下:
为了避免其他报文影响,内部测试使用字节1400报文。
从SR6604路由器 <SR6604>ping -s 1400 -c 100 -t 0 -a 192.168.0.207 10.26.108.1(192.168.0.207为SR6604路由器LOPBACK接口地址, 10.26.108.1为对端U200 防火墙LOOPBACK地址)。
PING 之前:
sa remaining duration (kilobytes/sec): 1837640/2978
PING之后:
sa remaining duration(kilobytes/sec):1837500/2949
1837640-1837500=140*1000=140000 正好对应100个 ----说明IPSEC加密正常
PFS: N, DH group: none
tunnel:
local address: 202.107.222.153
remote address: 122.228.178.206
flow:
sour addr: 192.168.0.0/255.255.0.0 port: 0 protocol: IP
dest addr: 10.26.0.0/255.255.0.0 port: 0 protocol: IP
[outbound ESP SAs]
spi: 0xC7F64900(3354806528)
transform: ESP-ENCRYPT-DES ESP-AUTH-MD5
in use setting: Tunnel
connection id: 6366922
sa duration (kilobytes/sec): 1843200/3600
sa remaining duration (kilobytes/sec): 1837640/2978
anti-replay detection: Enabled
anti-replay window size(counter based): 32
udp encapsulation used for nat traversal: N
原始IP报--------原始ICMP报文发送正常
*Jan 15 17:50:36:208 2016 SR6604-1 DPIPFWD/7/debug_case: -Slot=3;
Sending, interface = GigabitEthernet3/1/7, version = 4, headlen = 20, tos
pktlen = 1428, pktid = 5524, offset = 0, ttl = 255, protocol = 1,
checksum = 0, s = 192.168.0.207, d = 10.26.108.1
prompt: Sending the packet from local
IPSec加密后报文----------------------加密后封装隧道头ICMP发送正常
*Jan 15 17:50:36:208 2016 SR6604-1 DPIPFWD/7/debug_case: -Slot=3;
Sending, interface = GigabitEthernet3/1/7, version = 4, headlen = 20, tos
pktlen = 1480, pktid = 30552, offset = 0, ttl = 255, protocol = 50,
checksum = 0, s = 202.107.222.153, d = 122.228.178.206
prompt: Sending the packet from local
驱动--------------------驱动正确发送
*Jan 15 17:50:42:444 2016 SR6604-1 DRVDBG/7/debugging: -Slot=3;
(GigabitEthernet3/1/7)PHY/PKT:
Packet Output, Packet Len =1494,Partial data as follows:
-----------加上12字节以太头以及2字节前导码
00 E0 FC A4 B0 CD 70 F9 6D 1A 6B 4F 08 00 45 00
05 C8 92 64 00 00 FF 32 4C E7 CA 6B DE 99 7A E4
B2 CE CB 7C B6 3F 00 00 1B AD 86 BA 48 42 90 C5
72 85 9C FB D0 90 68 67 24 6C 8A 01 CD 32 07 D3
DIP :CA 6B DE 99 SIP:7A E4 B2 CE
从SR6604路由器上查看,加密数据的sa的remote address 是122.228.178.206,并非是U200上的任何接口地址。也就是说,从U200<U200>PING -c 10 -a 10.26.108.1 192.168.0.207 的回报文是转发给了122.228.178.206这个设备,经过确认,为Juniper路由器。
<SR6604>display ipsec tunnel | begin 10.26.
Dest addr :10.26.0.0/255.255.0.0 port:0 protocol: IP
Current Encrypt-card:
Connection id :139432
Perfect forward secrecy:
SA's SPI:
inbound: 187199087 (0xb286e6f) [ESP]
outbound: 3562274487 (0xd453feb7) [ESP]
tunnel:
local address: 202.107.222.153
remote address : 122.228.178.206
flow:
sour addr : 192.168.0.0/255.255.0.0 port: 0 protocol : IP
dest addr : 10.26.0.0/255.255.0.0 port: 0 protocol : IP
current Encrypt-card:
至此问题似乎已经比较明确了,但是,为什么到U200的流量,转发到了122.228.178.206 这台设备呢?再次确认后发现:
sour addr : 192.168.0.0/255.255.0.0 port: 0 protocol : IP
dest addr : 10.26.0.0/255.255.0.0 port: 0 protocol : IP------16位掩码,应该是32位才对。SR6604路由器IPSec采用的是模板方式,掩码信息是由IKE PEER协商出来的,也就是说,应该是Juniper发送的掩码信息是错误的。
最终确认是客户更改了Juniper设备的IPSec 安全ACL的网段掩码导致这次故障。