Solve[
Exp[-Lambda^2]*(alpha*t0^alpha*tau^(-alpha - 1))*(t0^
alpha (t - tau)^-alpha) == HatPsi[tau]*(\!\(
\*SubsuperscriptBox[\(\[Integral]\), \(t -
tau\), \(\[Infinity]\)]\(HatPsi[z] \[DifferentialD]z\)\))*\!\(
\*SubsuperscriptBox[\(\[Integral]\), \(t\), \(\[Infinity]\)]\(HatPsi[
z] \[DifferentialD]z\)\)/\!\(
\*SubsuperscriptBox[\(\[Integral]\), \(t\), \(\[Infinity]\)]\(Psi[
z] \[DifferentialD]z\)\), HatPsi[z], z]

正确的命令是
Solve[
Exp[-Lambda^2]*(alpha*t0^alpha*tau^(-alpha - 1))*(t0^
alpha (t - tau)^-alpha) == HatPsi[tau]*(\!\(
\*SubsuperscriptBox[\(\[Integral]\), \(t -
tau\), \(\[Infinity]\)]\(HatPsi[z] \[DifferentialD]z\)\))*\!\(
\*SubsuperscriptBox[\(\[Integral]\), \(t\), \(\[Infinity]\)]\(HatPsi[
z] \[DifferentialD]z\)\)/\!\(
\*SubsuperscriptBox[\(\[Integral]\), \(t\), \(\[Infinity]\)]\(Psi[
z] \[DifferentialD]z\)\), HatPsi[z]]

大概是积分方程难搞，想办法搞成微分方程试试？