failed to lazily initialize a collection of role

I am using Spring and Hibernate 3. The problem I am running into is lazy loading, the session is getting blasted when Flex tries to create an object that has sets.

<set name="loans" inverse="true" lazy="true">
<key>
<column name="amortization_type_id" />
</key>
<one-to-many class="com.gcloans.dao.hibernate3.Loan" />
</set>

I don't want to have lazy="false" for memory issues. Any ideas.