Troubleshooting & monitoring mongodb performance
Trouble: In our production systems we started to receive that the Apache current connection is above the threshold(6k) for few mins. It went up to a maximum of 17k from 1k in 5 minutes and then again it returned to 1k in about 10 mins. It's triggered by the spike in traffic from 500 req/sec to 1500 req/sec in under 2 mins. It's puzzling to us because it's usual that in our products that during this peak time, the traffic almost tipples for few mins and comes back to normal levels. But never caused system performance degradation.!! Architecture: Mongodb 3.2 in CentoOS 6. One primary, two secondary replication mode. Data profile: It hosts two db, one has app related config collections, other db has app related KPI logs collections. config - very low write, high read. Total data size is < 1GB KPI logs - constant write load from log forwarders(Fluentd), occasional read from reporting system. Total data size is around 500GB. Troubleshooting: I started...