I have never posted on this forum before but I often leverage the information available here to minimize my learning curve. I have not been able to find any solution to the issue below so I decided its time for my first post. Some of the information below has been regurgitated form the posts of others on this forum
At random time intervals ranging from once per hour, to once per 20 hours, I am having a “Get Image2.vi” timeout error -1074360293. This error is happening on multiple computers, in LabVIEW, in executables, in Max, and in the camera vendor provided software. I am running 4 different computers each with 2 cameras. The surprising thing is that 1 of the computers never has the error while the other 3 do. This is true even when I swap around the GigE cards, the cables, and the cameras. The PC without the error is ~2 years old and the 3 that are failing are newer Alienware 51 computers. I have been unable to make any sense of this information despite a significant effort.
I am using an Adlink GIE62 card with the Intel PRO chipset. The cameras are FLIR (formally Point Gray) BFLY-PGE-50S5M-C (5MP Sony CMOS).
Here is a list of all the things I have tried so far without success
- The software input value of “Get Image2” was changed from “Last New” to “Next”. This change made no noticeable, positive or negative impact to the primary issue but is presumed to be a minor, generic programing improvement. We will default to “Next” moving forward.
- GigE driver version was evaluated and was ruled out as the primary issue. Despite this fact, a significant amount of data does exist that proves a significant value add when using the NI “High-Performance Driver” instead of the Intel PRO/1000 default driver. In addition to reducing processor overhead, this driver eliminates the potential of windows firewall and other antivirus software from blocking video packets. This type of packet blocking can be intermittent, difficult to troubleshoot, and should be avoided by always using the NI “High-Performance Driver”.
- GigE cable quality was previously studied and has proven to offer value regarding EMI tolerance. Regarding the current issue, only the high-quality cables have been tested.
- Camera thermal variance was studied and ruled out as a contributing factor. In a short (~1 hour) test one camera was limited to 28°C with heatsinking and a second was elevated to 75°C using a heater. There was no noticeable performance difference between the two cameras. As a reference, a camera mounted in a CMS will typically level off ~45°C and a “floating” camera with no airflow (in a bag) will typically be ~65°C
- Camera settings
- Firewall Traversal attribute is Enabled. (This is not believed to be relevant when using the NI “High-Performance Driver” but is good practice.
- Camera Timeout values were not studied during this analysis, but it is believed that a value of ≥1000 will minimize the likelihood of false errors. This is important when the PC momentarily becomes distracted by other processes (like network communication) especially if the PC is in any form of hibernation. This includes when the processor has throttled down to save power (windows default).
- Windows power management has been set to “performance mode” to prevent the CPU from “throttling down” or attempting to sleep in any way.
- Jumbo Packet size is set to a value of 9000 to provide the best performance and minimize PC overhead.
- Test Packet Enabled is set to disabled. This was studied and circumstantially ruled out as the primary cause of our camera crashing issue. Despite this fact, there is historical Forum data on the showing that having test packets enabled can cause the exact symptoms we are experiencing. This change is known to potentially cause a new error of 0xBFF60493 which is fixed by turning off the option to "Display images on remote monitor." (In MAX (Tools->NI Vision->Remote Image Options...)
- Packet Resends Enabled is set to disabled. The supporting logic for this change was not directly found on the NI forum but instead was formulated from multiple separate pieces of information including:
- Since image data packets are streamed using the UDP protocol, there is no protocol level handshaking to guarantee packet delivery. Therefore, the GigE Vision standard implements a packet recovery process to ensure that images have no missing data.
- The GigE Vision header, which is part of the UDP packet, contains the image number, packet number, and timestamp. As packets arrive over the network, the driver transfers the image data within the packet to user memory. When the driver detects that a packet has arrived out of sequence (based on the packet number), it places the packet in kernel mode memory. All subsequent packets are placed in kernel memory until the missing packet arrives. If the missing packet does not arrive within a user-defined time, the driver transmits a resend request for that packet. The driver transfers the packets from kernel memory to user memory when all missing packets have arrived.
- CPU Usage: Unlike other machine vision bus technologies, which can DMA images directly into memory, Gigabit Ethernet requires CPU usage for packet handling. This means that you are left will fewer CPU cycles to perform image analysis. While CPU usage can be mitigated to a certain extent by using the high-performance driver, it cannot be eliminated.
- Latency: Because network packets can get lost, requiring resends and network packet handling is CPU dependent, there is a non-trivial latency between the time a camera captures an image and the time it appears in user memory.
I have attached a simplified version of code that creates the error and logs the error information to a file. As I said earlier, I don’t expect the code to be helpful since the issue also happens in Max and the camera vendor provided SDK software.
Despite all of the above, the issue persists and I have no idea what to try next.
If you have any insight or know of any ideas I can test, please reply.