Setup Splunk Forwarder to monitor csv file

This post is basically a walkthrough about how to setup Splunk forwarder and configure it to monitor csv files. It was reasonably difficult for me to collect proper information on this topic as splunk documentation is not very extensive in terms of how it can be configured with different data sources that it supports. I charted out this information from answers to questions asked in splunk forums.

For understanding on ‘Splunk Forwarder’, please visit this link.

Now, let’s kick the walkthrough.

 STEP 1 – Setup Splunk web to receive input from Forwarder
1) Login to the splunk web. Go to Settings -> Data and click on ‘Forwarding and Receiving’.
2) Click on ‘Add New’ link to add new receiving and set it up to listen to port # 9997. Click ‘Save’.

STEP 2 – Download and Install Splunk Forwarder
1) Download and Install splunk forwarder. Please note during installation, in step 5, mention port number of Receiving indexer as 9997.

STEP 3 – Modify the Configuration files in Splunk forwarder to monitor csv file
1) Add the stanza shown below to INPUTS.CONF file which can be found at $splunkforwarderhome\etc\system\local

[monitor://C:\blah\Somecsv.csv]
sourcetype = CustomParser

Please note the ‘sourcetype’ here. ‘CustomParser’ is the source type that will be created in Splunk Receiver. More explanation will be followed in the step that creates this source type.

STEP 4 – Modify the configuration files in Splunk Receiver
1)  Modify PROPS.CONF file seen at $splunkhome\etc\system\local with the code shown below. If this file doesn’t exist at this path then manually create one.
[CustomParser]
SHOULD_LINEMERGE = False
REPORT-r1= ReportCsv

This creates a sourcetype ‘CustomParser’ which is used in Step 3. ‘ReportCsv’ is something that specifies how fields will be extracted from your csv and is explained in next step in this section.

2) Add the stanza shown below to the file TRANSFORMS.CONF found at $splunkhome\etc\system\local. If the file doesn’t exist, create one. Modifying any file under $splunkhome\etc\system\default is not recommended.

[ReportCsv]
DELIMS=”,”
FIELDS=”Id”,”Country”,”State”

Here replace FIELDS with fields from your csv file. These are the field names that should be used in your splunk queries. Example splunk query – Yourhostname | table Country   will give you values of ‘Country’ field.

3)  Restart both Splunk Forwarder and Splunkd services and enjoy searching !!

Trackbacks

Leave a Comment

Your email address will not be published. Required fields are marked *