![]() Once they leave the wire the listening application must handle them appropriately or they are lost. Network requests are, by design, ephemeral. Put simply, traffic replay is the ability to capture computer network traffic and recreate it in its original form. When the application receives the stream in very quick succession, it may re-assemble the data with its underlying TCP/IP stack.Traffic replay is quickly gaining traction as the best way to recreate production scenarios. Even if the original packets (layer 3) are fragmented, since the layer 4 responsibility was delegated to net/http, no matter how many individual conn.Write() we do, it will be treated as stream. The delay is very important if you need to recreate the fragmentation. More information about the filtering syntax can be found at this link: Sending Packets ![]() The same can be said about the unneeded side of bi-directional traffic. In replaying, they don't have much use and will be considered noise. PCAP files used in debugging usually have broad capture filters, as the other captured information can be as useful as the main target protocol. ![]() This is also used in ngrep and other packet capture tools. Loading PCAP FileīPF is very handful when dealing with filtering network traffic. With Go, this has been very easy with the help of net/http and /google/gopacket libraries. Without the need to re-implement the protocol in the server side, the most promisying way is to read the PCAP file, and send it to the wire (on whoever is connected to the dummy socket). Developing/extending a simulator (although we do have one) to do this is one of the options, but due to the urgency of the problem, a much quicker solution is more favourable. The packet capture step was very helpful at this stage in providing the packets in PCAP format, but we needed a way to re-send them into the application without connecting to the actual server in production. The general idea that was concluded when evaluating the validation step, is to re-produce the offending packets. Preferences > Protocols > Re-using Production Traffic Quick Tip: You can turn off TCP re-assembly in Wireshark to see fragmented packets individually. Wireshark of course helps me navigate and play with the PCAP file and see the problem in a beautifully parsed presentation. My favorite tool for this task is ngrep, which can do things like normal grep on the network layer, and the ability to save the captured packets into a PCAP format. This of course adds strain to the system load and puts the interface in promiscuous mode, so it's not something that should be turned on permanently. One of the little weird joys I do while debugging live (network-oriented) production issues, is to look into the network layer and see the packets coming in. But thanks to a couple of handful experiences (failures included), this is out of the question anymore. The younger version of myself would think to test the fix in production, where it can easily be replicated. The issue was fairly trivial in nature, and could've been prevented with certain measures, but that's not what this post is about. It turns out our PDU parser written in Erlang was failing to recognize packets fragmented throughout the network. Upon further digging, it seems that the volume wasn't the main contributor for the problem. This is quite unusual since all of our other connections were stable, despite the huge traffic we're getting and generating from time to time. On a very recent set of events in my workplace, we discovered we were having problems receiving server-initiated traffic from one of our connectivity partners.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |