What's the difference of AC reactor against DC reactor in inverter?

AC reactors reduce harmonics, mitigates transient conditions and helps to maintain proper phase to phase voltages. DC reactors will also mitigate harmonics however it will provide more harmonic attenuation than a equally sized AC reactor.

The DC reactor will reduce current transients through the rectifier that are resultant from voltage transients. A small voltage transient will cause a large current transient/surge if no reactor is present, so the DC reactor does provide protection against transients causing rectifier failure due to current transients, but the AC reactor provides additional protection in that if the loaded transient voltage is high enough to cause a reverse voltage of sufficient amplitude to cause the rectifier to fail due to over voltage rather than over current, then the additional voltage drop across the reactor will reduce the voltage across the rectifier.
Up to this level, both the AC reactor and the DC reactor will provide increased immunity to transients and surges.
The cost of the DC reactor (in manufacturing volumes as opposed to one off specials) is generally lower, less copper and less iron, and the power loss (heat) in the DC reactor is also lower.
Frequency inverter with a built in DC reactor will perform much the same as one with a built in AC reactor with an advantage of size, price and efficiency, but a disadvantage of a lower peak line voltage threshold.
As external add ons, the AC reactor is usually a more commercial solution.

Both reactors will increase the conduction angle of the DC current pulses flowing through the rectifiers and thereby reduce the harmonics on the supply.
The AC line reactor will provide a high impedance path for fast transients and reduce the transient voltages across the rectifiers providing a level of transient voltage protection for the input rectifier.
Both reactors will reduce the charging current to the capacitor bank under a voltage surge condition, so help to protect the rectifier from current surges resulting from voltage surges/transients.
The AC line reactor has an AC flux in the iron, with a high harmonic content, and so has both copper loss and iron loss.
The DC bus choke has DC with a superimposed ripple current, so the iron losses are lower.

Major differences? AC has higher losses but improved transient voltage protection for the rectifier. DC has lower losses, so higher efficiency.

Very important is that most DC bus chokes will need to be custom designed for the drive as they are commonly mounted inside the inverter chassis. (and frankly, I can't think of any reason to use a bus choke over a line reactor if both would have to be externally mounted, the internal mount is the only real advantage to the bus choke). Thus while there may be less weight for a bus choke, it's cost can be higher. On the other hand, multiple manufacturers have a wide range of ratings for 3 phase line reactors. To the point where the 3 phase reactor is practically a commodity and thus can be obtained very quickly and inexpensively. This is also important if a replacement is ever needed.True in most cases the reactor will need to be mounted external to the drive chassis, but this is offset by the cost savings of buying the reactor direct from a distributor. And not paying double markup necessary when it comes in the drive from the frequency inverter OEM. Also, since the BUS choke is custom, there will be little option for varying the inductance and thus weighing cost vs performance for different applications. Reactors are commonly available in 1.5%, 3%, 5% and 10% impedance ratings.