Remove duplicate events from splunk

To remove duplicate events from splunk, follow these steps –

Step 1 – Put all the duplicate events in lookup table.
index=* sourcetype=[SourceType] | eval id=_cd.”|”.index.”|”.splunk_server | transaction _raw maxspan=1s keepevicted=true mvlist=t | search eventcount>1 | eval delete_id=mvindex(id, 1, -1) | stats c by delete_id | outputlookup delete_these.csv

Step 2 – View the events stored in lookup table
| inputlookup delete_these.csv

Step 3 – delete those events from actual source type which also exists in lookup table
index=* sourcetype=[SourceType] | eval delete_id=_cd.”|”.index.”|”.splunk_server | search [|inputlookup delete_these.csv | fields delete_id | format “(” “(” “OR” “)” “OR” “)”] | delete

Reference – http://answers.splunk.com/answers/69924/how-to-delete-duplicate-events