In decode-and-forward (DF) MIMO two-way relay systems, the transmission schemes are designed and the closed-form expressions for the outage probability and average symbol error rate (ASER) of the twoway relay system are derived based on two different scenarios of channel state information (CSI). For perfect CSI, the maximum-ratio-transmission and combining (MRT-MRC) technique is applied to design the beamforming and combining vectors. Without perfect CSI, the transmission scheme with limited feedback is designed, and the analytical results are verified through two kinds of codebooks, i.e., random vector quantization and Grassmann. The simulation results show that, the proposed transmission schemes for the two-way relay system can outperform other transmission schemes in the performance of outage probability and ASER, and the accuracy of the derived closed-form expressions is also verified by the numerical simulations.
