Can I plant my cactus outside?

I have a small, maybe 5 inch tall fairy castle cactus in a pot. Would it live if I planted it outside in my yard? I live in New Jersey.