In Progress

NPM Module to export time series data to AWS Glacier and S3

We currently have a MySQL database table that holds about 2 TB of data in approximately 9 billion rows. Each row contains one second of power data from an electricity meter, with 31 columns in each row. The data is indexed on a date-time, a controller id and a power meter id.

The database is a production database, and the table is constantly being written to. We intend to create a read replica of the database, and then break the link between the read replica and the master. From the read replica, we'll be able to dump the data to AWS Glacier without impacting our production system. However, we would like to put in place a service that will keep this table to a more manageable size.

Service Requirements:

* The service should be written in Node JS >= v5.0.0.

* The service should comply with the AirBnB style guide.

* The service should be written as an NPM module, i.e. it can be required into another Node JS application.

* The service should not severely impact the production database, i.e. it should minimise the number of connections & resources required on the database, it should not try to pull all the data in one query. The service can take its time to complete.

* The service should only archive data that is over a year old.

* For data that is between 1 and 2 years old, the service should archive it to S3.

* For data that is over 2 years old, it should archive it to Glacier.

* The service should also scan S3 for files that are older than 2 years and migrate them to Glacier.

* The archived data should be stored in CSV files, with each file containing the data for one controller id and one meter id, with the starting date-time, controller id and meter id in the filename.

* Each file should contain one week of data.

* The service must be able to restart from where it left off if it is restarted, losses connection to the database etc.

* The completed project must contain tests.

Project Requirements:

* Applicants should be able to provide a outline of how they will implement their solution.

* All code to be stored in a private git repo provided by us.

* The service can use the AWS SDK and any available AWS resources.

Sample data will be made available for development and testing purposes.

Applicants should provide a cost and time estimate and examples of their previous work.

Skills: Amazon Web Services, MySQL, node.js, Software Architecture, Software Testing

See more: time series data modeling, time series display data, mysql time series data procedures, matlab time series neural network test data, time series data econometrics project, time series data clustering matlab, module export data xml drupal, forex data time series, data time series, data entry online html need store mysql database, time series data minitab, upload data mysql database table vb net, notify client data change mysql database table wcf, insert excel data mysql database table file jsp hibernate, data fetch mysql database table vb net, retrieve data mysql database table scroll, estimating time series data multiple time series, time series data library, time series analysis project, export eagle point data hec hms, software time series analysis hurst exponent

About the Employer:
( 0 reviews ) Limerick, Ireland

Project ID: #17213799

Awarded to:

PaoloBarbolini

Hi, I have 3 years experience with aws. The way I will build it is I think I will make it create csv files every hour, upload them to a temporary s3 bucked and then when there are one week of csv files we merge all More

€150 EUR in 3 days
(2 Reviews)
1.5

7 freelancers are bidding on average €192 for this job

€155 EUR in 1 day
(55 Reviews)
6.2
nikosku

Hello my name is Nikos and Im working on the Linux server administration field for the past 6 years. Over these years I worked for two web hosting companies as a Senior Administrator managing their servers & providi More

€155 EUR in 1 day
(49 Reviews)
5.8
dirisalagopal

amazon web services expert.

€250 EUR in 1 day
(18 Reviews)
4.4
hybridskill

I am an AWS Certified Solutions Architect and have migrated over 100 projects to the cloud with best practices.

€111 EUR in 2 days
(1 Review)
0.5
€244 EUR in 3 days
(0 Reviews)
0.0
helloprasad009

Hello Sir/ Madam, I am Shiva Prasad and i feel that i would be a perfect fit for this project. let me explain the reason why i would be a strong choice. I am a active and keen learner of all the opensource technolog More

€277 EUR in 5 days
(0 Reviews)
0.0