Upstream jitter

We were having issues with phone calls earlier. We use a VOIP system and contacted the provider who had us a run some tests. According to the tests most everything seemed fine except for the upstream jitter was way out of wack ranging from 200ms-320ms each time we tested. Also were having packet loss of 2-4%. The phones seem to be working ok again but the upstream jitter on the tests still seem to be around 50ms, but the downstream jitter is fine around 3ms. Packet loss is still occurring according to these tests. Any help would be appreciated.