Sieging a castle in fall of the samurai is significantly harder than doing it in vanilla. Troop losses expected are 1000 men minimum at the beginning (vs 1-3 garrisons of line infantry). Unlike in vanilla , the ai can't be destroyed while enclosed in their walls. As such , even with 3,000 rifle units firing at a wall of line infantry , you can expect a minimum of 600 losses when illing about 300 enemy or a 2:1 ratio.
As such there is a simple solution. simply recruit traditional bow kachi. You will save your men and whoever you plan on rushing in afterwards thousands of units.
When you finally get artillery, you can turn everything around much more quickly and blow castle wall defenders to bits.