On the basis of the point charge model, we formulate the CEF Hamiltonian H_CEF in the rare-earth based QC and approximant crystal (AC) with ligand ions located at pseudo 5-fold configurations by using the operator equivalent method. By setting the total angular momentum J=7/2, the CEF in the quantum critical QC Au₅₁Al₃₄Yb₁₅ and the 1/1 AC Au₅₁Al₃₅Yb₁₄ is analyzed with consideration for the effect of Al/Au mixed sites. We find that the ratio of the valences of ligand ions α=Z_Al/Z_Au plays an important role in characterizing the CEF ground state. As x decreases from α=3, the 4f wave function of the CEF ground state with the flat shape lying in the mirror plane is deformed around α≈0.8 to the flat shape perpendicular to the pseudo 5-fold axis at α=0. The formulated H_CEF by J is generally applicable to rare-earth-based QCs and ACs, which is useful to analyze the CEF.