Is anyone familiar with or have experience using harmonic mitigating transformers to eliminate 5&7 order harmonics on line side of vfd

Funny! I have extensive experience in exactly that (the topic of my PhD thesis). In my case I controlled the 5th and 7th order harmonics so to mitigate torque ripple in 3-phase motors.
Regardless, if it is for a motor or a transformer, conceptually it is very similar.
Please feel free to contact me if you have any questions.

Use Dy transformer on supply of VSD. The angle of 5th harmonics is 120 degrees and 7 th harmonics 180 degrees ( secondary leads 30 degrees on primary), when using connection group Dy5, harmonics from VSD are 180 opposite to other harmonics on trafos primary side and these will compensate each other. This is case by case study and You have to know characteristic of supply grid very well.

When vsd is not run at full load measured THD will be high. much better if you run your vsd at full load, base on my experience, 6-pulse vsd will be difficult to reach limitation of THD as regulated by IEEE 519 although it is run at full load. I ever succeeded when run vsd with low load of VSD (163 kva vsd was loaded with 38hp motor). I got TPF value 0.98 and ITHD is on IEEE range. A leasson learned that I got, focused on notching process and displacement of power factor when gate of scr is switching had helped me to find the solution.