https://gcn.com/articles/2019/08/01/nyc-cloud-cyber-pipeline.aspx
By Stephanie Kanowitz
GCN.com
August 01, 2019
Two years after New York City Mayor Bill de Blasio created the NYC Cyber
Command to lead the Big Apple’s cybersecurity defense efforts, the team built
an open-source, cloud-based data pipeline to serve as a security log
aggregation platform that analysts could use to quickly detect and mitigate
threats to city networks and systems.
In accordance with its cloud-first strategy, NYC Cyber Command built the
pipeline on Google Cloud Platform (GCP) and Google products such as Cloud
Pub/Sub, a scalable data analytics product that facilitates data ingestion.
Security events are published to Cloud Pub/Sub and then pull subscriptions make
the data available to log parsers and other services via Google’s Cloud
Dataflow, a fully managed service for stream and batch processing that puts the
data in formats security analysts can use.
“We have data coming from external vendors, and all this data is ingested
through Pub/Sub, and Pub/Sub pushes it through to Dataflow, which can parse or
enrich the data,” said Noam Dorogoyer, a data engineer and IT project
specialist at the command. “The way the data comes in can be simple such as
comma-separated. Other times it’s a mess. There is not a common format among
the vendors.”
The command uses logic in Dataflow to move the data into BigQuery, Google’s
serverless cloud data warehouse, which puts it into a tabular format that’s
easy for analysts to access.
[...]
--
Subscribe to InfoSec News
https://www.infosecnews.org/subscribe-to-infosec-news/
https://twitter.com/infosecnews_
