Could you please share with me the model number of the this camera? What is the resolution this camera is outputting to the AVioHD?
VLC, while it’s a great “does many things” application, it’s best to specify the exact video specifications (resolution, aspect ratio) when capturing video with it to ensure the ingested video isn’t being scaled in an undesirable way. You could alternately try OBS, AMcap or even Windows Camera to see if the quality is as poor in those applications as well.
What may be occurring, is that the camera is outputting a lower resolution while the capture program (VLC) is requesting a much higher resolution (ex. 480p vs 1080p). In this case, the AV.ioHD would rescale the image from the source to match what the capture program is requesting. If the source is a much smaller resolution then requested by VLC, the imagery being captured will indeed look pixelated. This would be very similar to making a lower quality digital image larger; the larger you go, the more pixels you will see.