Log Processing Exercise

  • Log-in into your Elasticsearch sandbox
  • Make sure elasticsearch is running:
    curl localhost:9200/_cluster/health?pretty
    
  • Create configuration file:
    sudo nano /etc/logstash/conf.d/cloud-init.conf
    
  • Copy-paste settings we have reviewed during previous slide:

    input {
      file {
        path => "/var/log/cloud-init.log"
        start_position => "beginning"
        type => "logs"
      }
    }
    
    filter {
      grok {
        match=> {
          "message"=>"%{TIMESTAMP_ISO8601:datetime}%{SPACE}%{SPACE}-%{SPACE} (?<module>(?<= - )(.+)(?=\[))(\[)(?<loglevel>(.+)(?=\]))(\]: )%{GREEDYDATA:message}"
        }
      }
    }
    
    output {
      elasticsearch {
        hosts => ["localhost:9200"]
        index => "cloud-init"
      }
    }
    
  • Logstash loads all files in the /etc/logstash/conf.d directory, store no extra files there
  • Restart logstash service and monitor messages:
    sudo service logstash start && sudo tail -f -n 100 /var/log/logstash/logstash-plain.log &
    
  • Check the data has been populated into elastic search:
    curl localhost:9200/cloud-init/_search?pretty=true
    
  • Expected results is a long list of parsed log events
  • New lines added to the log file will be posted into the index

results matching ""

    No results matching ""