Abstract: We report our recent progress on the lattice calculation of the hadronic light-by-light (HLbL) contribution muon $g-2$. Previously, we have completed the connected diagrams and the leading disconnected diagrams calculation on the $48^3 \times 96$ lattice with physical pion mass and 5.5 fm box size. The result is $a_\mu = 5.35(1.35)\times 10^{-10}$, where the error is statistical only. We expected sizable finite volume errors, mostly from the QED part of the calculation. To address this issue, we developed a strategy which allows the QED part of the HLbL diagram being evaluated separately in the infinite volume with continuum propagators. We have obtained some preliminary results on RBC/UKQCD's recently generated coarse $48^3\times 64$ lattice with physical pion mass and 9.6 fm box size.