RSS FEED FEEDER FREE DOWNLOAD INSTALL

Further research led us to Alexander Marquardt ’s blog, Deduplicating documents in Elasticsearch, which in turn led us to Elastic’s fingerprint filter plugin - a much simpler and equally effective solution. We initially wrote a Python script for data deduplication, which worked but seemed excessive.

We would be like, “Cool! A document referencing a Zero Day!” and then an hour later we were like, “Sweet! This is the same article we saw an hour ago!” It was ugly, and we knew it would negatively impact our analysis and research as we moved forward. We would sporadically get one new document and several duplicates. Logstash would pull from the RSS feeds every hour, as per our configuration. We quickly determined that Logstash filtering would be necessary in order to avoid ingesting duplicate documents.

RSS FEED FEEDER FREE DOWNLOAD INSTALL

It can be installed by running bin/logstash-plugin install logstash-input-rss. Note: The RSS plugin needs to be installed as it does not come as part of the logstash package.

Again, we wanted our tags to be succinct so that we could clearly differentiate the news feeds we chose. We are using tags for Kibana filtering purposes.

tags : Add any number of arbitrary tags to your event.

We opted to set the interval to 3600 seconds (1 hour) our decision was purely based on preference.

interval : Interval to run the command.

We chose to use multiple RSS inputs as shown in the image above. We also set up our ids with clear, distinct names, all of which we found to be especially helpful during troubleshooting and testing. If you have multiple RSS inputs, adding a named ID will make it easier to monitor Logstash. This is particularly useful when you have two or more plugins of the same type.

id : A unique ID to the plugin configuration.

RSS input plugin : Within the plugins input section, we added the following: Here is an overview of our Logstash pipeline and its configuration. Our project’s name is A Quick RSS Cybersecurity News Feed. At that point, we had identified all of the necessary ingredients to start building an Elastic-based Open Source Intelligence Tool (OSINT). Our initial impulse was, “let’s write a Python script, send the data to Logstash, watch it flow into Elasticsearch, and observe via Kibana…” As we started planning and researching, we encountered a solution that would allow us to bypass the Python script entirely: the RSS input plugin for the Logstash pipeline. Many feeders and aggregators are subscription based, but we feel that content curation should be free.

There are several ways to consolidate RSS feeds. We realized that we were spending a lot of time visiting the same sites and needed to consolidate the feeds (duh!).īelow, we’ll break down how we used RSS feeds, Logstash, Elasticsearch, and Kibana to ingest, consolidate, aggregate, visualize, and search cybersecurity content of interest - and how you can do the same. We are constantly researching - sifting through news articles, white papers, databases, etc. We are always looking for ways to leverage the Elastic Stack to add value to the Elastic and cybersecurity community. As former SOC analysts, we find ourselves drawn to the security use cases. As Elasticians, we have the opportunity to observe various Elastic use cases.

Rss feed feeder free download