I know nearly nothing about your situation, but I've encountered similar problems before so I'll just tell you what I've generally found to be the culprit in such a scenario.

VoIP (especially the multi-user variant) suffers greatly from network congestion. If your network is operating sub-optimally, this may be the reason you are experiencing call drops. Again, I don't know anything about your network's structure, but some gateways have a QoS setting that you can toggle to prioritize VoIP traffic, so this may be an option to consider as it is a very easy change to make.

You may also want to check your firewall rules, as sometimes a firewall not properly configured for VoIP will drop subsequent datagrams if one related to the initial one hasn't been sent within the (usually quite small) timeout period. Especially on a complex network with sketchy routing, increasing this timeout value may yield the results you are looking for.

There's also the possibility that the problem isn't on your end. The same issues I've described above can occur on any network, so the person on the other end may be initiating the call drop instead.