DSolve[{(M (y'[t] y[t]/Sqrt[(x^2)[t] + (y^2)[t]] +
x'[t] x[t]/Sqrt[(x^2)[t] + (y^2)[t]])' + g)*-x[t]/
Sqrt[(x^2)[t] + (y^2)[t]] ==
m x''[t], (M (y' [t] y[t]/Sqrt[(x^2)[t] + (y^2)[t]] +
x'[t] x[t]/Sqrt[(x^2)[t] + (y^2)[t]])' + g)*-y[t]/
Sqrt[(x^2)[t] + (y^2)[t]] == m x''[t], x[0] == 0, y[0] == y0,
x'[0] == vx0, y'[0] == 0}, {x[t], y[t]}, t]

x[t]^2 这样写才对

DSolve[{(M*
D[(y'[t] y[t]/Sqrt[x[t]^2 + y[t]^2] +
x'[t] x[t]/Sqrt[x[t]^2 + y[t]^2]), t] + g)*-x[t]/
Sqrt[x[t]^2 + y[t]^2] ==
m x''[t], (M*
D[(y'[t] y[t]/Sqrt[x[t]^2 + y[t]^2] +
x'[t] x[t]/Sqrt[x[t]^2 + y[t]^2]), t] + g)*-y[t]/
Sqrt[x[t]^2 + y[t]^2] == m x''[t], x[0] == 0, y[0] == y0,
x'[0] == vx0, y'[0] == 1}, {x[t], y[t]}, t]

第一个和楼上说得一样，那个是你报错的原因，但是我改了一下，没有报错，可能是DSolve解不出来这个方程

又在算椭圆轨道是吧：
tieba.baidu.com/p/7874713620