Before you can connect your PlayStation Vita to your PlayStation 3 please check:
In some instances unplugging the USB cable from both the PlayStation Vita and the PayStation 3 and then plugging it back in can help.
To connect your PlayStation Vita to a PayStation 3 and perform operations such as copying data, the PayStation 3 System Software must be version 4.00 or higher. Find out more about the PlayStation 3’s System Software Updates.
If a game or another feature (including PlayStation Netowrk functions such as Account Management) is being used, you'll be unable to connect your PlayStation Vita to the PayStation 3. To connect your PlayStation Vita to the PayStation 3, please make sure that the PayStation 3 system is at the XMB.
Before you can transfer content between your PlayStation Vita and PayStation 3, you'll need to have a Memory Card inside the PlayStation Vita.
The same account must be signed in on both devices before you can transfer content.