You're in LA?

Then you can find small lava rock (cinder, scoria) at Sunburst in Irwindale. It's about $110/cuyd.

I don't have a lot of experience with growing in ground, but IME you can't break up small particles by adding larger particles. I once tried mixing my own clay soil with at least as much granite for a succulent raised bed, but it didn't work well. The smaller particles simply fill the spaces between the larger ones. It's like how chunky peanut butter is still just as thick as regular. I think your best bet would be to build a raised bed and fill it with new soil, including lots of grit and no clay.

-Chris

A girlfriend of mine has worked 1/3 course sand , !/3 pea gravel into 1/3 native soil ( a black clay gumbo) then she adds a bit of compost, not much into areas that have plants.. She often just mounded up the sand and gravel on top with rocks and then worked in some compost. She also worked in in areas. She has done this with her entire lot. She had to put in dome barriers to hold back the run off of soil during rains,and it did finally settle down and stay put. But she found that the gumbo , once broken down with 2/3rd material became a very fertile mix for cactus. She is in OKlahoma City so she is growing cylindropuntias, opuntias, cold hardy echinocereus and natives from the cold deserts and prairies.

I would imagine that scoria would be great, but one could do it with a washed decomposed granite sand and crushed granite. The crushed granite has a rough surface compared to the pea gravel and would capture air in the mix. My friend used sugar sand with is not recommended. Course washed contractors sand is preferred over sugar sand. I am using Washed decomposed granite sand and crushed granite with compost mixed into a caliche based native soil. It is not as fertile as her mix.

I do think that the clay needs more that 100% of sand/gravel added to it to break it up into a good draining friable. One might also ad some gypsum to it to facilitate in breaking up the clay. Can you get ahold of expanded shale , that is often used in Texas to break up clay. I think Pumice and scoria does the same. I do not like adding perlite into the ground. Stick with rock and sand. Keep adding till it is friable when you squeeze. Add some bone meal too. Greensand is an additive recommended for helping with clay and fertility. Talk with a KNOWLEDGABLE nursery that deals with bulk soil and haas people who know their stuff unhand. Big Boxes should be avoided.

I would talk to people who you see growing cactus in ground in your area and ask them what they did to modify it.Gardeners love to talk (if they were involved in the doing and not handed off to professionals) . Find what is available in your area in BULK amounts. Where did they buy their materials. How did they have it delivered or did they bring it in themselves. Did they bring it in by the truck load , trailers, bag by bag. I have been waiting for DH to rewire the trailer and I have been taking large 10 and 15 gallon pots and shoveling granite sand and crushed granite into it . I am loosing patience.., but my cactus bed is growing, just not fast enough!!!! I am still too much of a cheapskate to pay the extra for delivery, but I might do it because the other way is getting old and the size of my plans are bigger than my car. Poor little pontiac vibe.