Never tried oleander. I have added wisteria and rhododendron leaves to my compost pile, which became worm food. I noticed no ill effects. They are considered "toxic". It was a small percentage of the total compost pile.
If using them straight in the worm bin, I would suggest proceeding with caution. Experiment a little and let us know the results. Maybe start a small 1 gallon wormery with a few worms and a handful of oleander on top. I suspect all will be fine.

Good luck and happy wormin'

Pete

Yes, please keep us posted.
I have a great supply of Rhodo and Wisteria leaves but I have stopped using them as worm food, reading that they are toxic. I had added fresh shredded wisteria leaves several years ago and I noticed a decline in worm population. It was in late fall, and I didn't know about the toxicity of Wisteria or Rhodo for that matter. The decline of worm population could be caused by the cold weather, who knows.
The reason for not using Rhodo leaves, they are so strong and takes a long time to break down and I have other worm food more readily to be consumed.

I just situated my worm bin in the shade of an oleander plant. A few leaves got into my bin and I wonder if they would kill my worms.