Analysts and BI practitioners now have an intuitive solution that directly exports BigQuery, PostgreSQL, Adobe Analytics, and other marketing and analytics sources to Amazon Simple Storage Service (Amazon S3).

You may already be dumping multiple data sources in BigQuery, exporting them in a tabular format, and then uploading them into Amazon S3 to generate custom analyses. But this approach requires both engineering work and quite a few manual steps.

We built the Cognetik Data Streams connector for Amazon S3 to easily overcome that time-consuming and fragile approach.

With Data Streams, raw source data is sent directly to your S3 account, either in .json or .csv format, turning all this work into a simple-to-use interface.

The image below shows a stream configuration that migrates BigQuery data to Amazon S3.

 

Below is an example of a .json file created with Data Streams. In Data Streams, when naming the stream, the format is {prefix} – {granularity} – {start date}. No name gives the stream an ID. In the example below, we gave the file the prefix ‘json’ since we created it to exemplify the end result.

Cool Facts on Building Our 9th Destination Connector for Data Streams

While you can read the full release notes here, we thought it might be interesting to find out a few cool facts.

  • Amazon S3 supports multiple formats, so this was the first destination connector we built that allows the migration of data in multiple formats. Current release supports .json & .csv.

  • The S3 API does not have an easy way to append a created file, unlike other APIs we tackled in the past. For example, once a target destination would be set in a bucket, the second day would overwrite the file from the previous day. However, our engineering team implemented a solution that successfully updates day 2 data in a day 1 data file, without losing day 1.

 

To make the most of Amazon S3, head over to our Data Streams page to easily extract and load your data to AWS S3. 

Read here on how to get started . Or read more about the benefits of Amazon S3. 

 

About the author

Cognetik