android - fragments flow managing

I have a special flow of fragments in my app, so I need to be able to switch to any fragment, while keeping fragments in memory whenever possible (so if it's tight on memory, it's ok that the fragment will be released).

So far, I've succeeded doing a replace of the current fragment, but the thing is that the previous fragment is always being destroyed, so if I go back to it (using the action bar, for example), it's re-created and that takes some time.

The reason I use fragments instead of activities is the nice usage of the action bar, the ability to put multiple fragments inside the same container, the non-flexible activities-intents usage, etc...

The reason why I don't use the "Back" stack is that I wish to go to any fragment from any fragment, since the flow can change.

Here's a snippet of my code:

Fragment fragment=... ; //get the fragment from cache or create if not available yet...
 FragmentTransaction ft = getSupportFragmentManager().beginTransaction();
         ft.replace(R.id.mainActivity_fragmentContainer, fragment).commit();
 

I have also tried to use ViewPager, but since I don't want to have the sliding effect (which allows you to slide to other fragments) and since one of the fragments already include a viewPager, it's an obstacle. Disabling the sliding effect on the main viewPager somehow disables it on the other one.

BTW, I'm using the android support library for the fragments and not the native API.

my question is:
how can i achieve full control of which fragment to go to , while maximizing speed and avoiding too much memory being used for fragments ?

---

EDIT:
for now , i use the next workaround :
for the onDestroyView , i take the parent of the created view and remove the created view from there .

for the onCreateView , if i already have the created view from before , i return it .

however , i think it's a very risky thing to do , since i'm not sure of how fragments managing work , so it might cause weird problems . plus ,i'm not sure what will happen if android decides that it has low memory - will it destroy unused fragments (which is good) or will it cause out-of-memory exceptions (which is bad) .

---