President Joe Biden's inner circle is shrinking and now mainly includes family members and loyalists, The New York Times reported.