Closed

Import 10 million records 2GB .csv (5 parts) every hour to MySQL

Import .csv files from here to MySQL database:

[url removed, login to view]

Must be fast, easy to run, cheap, complete the process in under 5mins. It updates the CSVs' every hour and makes note of the data that changes.

What kind of server/mysql setup is needed on AWS? Let me know your estimations so I can judge how suitable you are for the job.

Skills: Big Data, Excel, MySQL, PHP, Software Architecture

See more: desktop software sort csv file million records, mysql csv import escape, import mysql csv perl, linux shell script mysql csv import, software write mq4, software write chip epson, useful software write book, import text mysql csv, unable add records mysql csv, software write web specs, perl mysql csv import, import csv access update records vbnet, mysql csv export import, vbnet import csv access existing records, free software write user guide, software write edid, free software write company profile, mysql csv import joomla, software write websites idea, mysql million records, mysql csv records, software write book images, software write books, mysql csv import perl, software write protection

About the Employer:
( 31 reviews ) London, United Kingdom

Project ID: #11724502

27 freelancers are bidding on average $160 for this job

aaravsoft

Hello, I will provide you desktop application for window where it will download csv file from companieshouse website and update the changes on your mysql database which few minute. You set interval time manually ( More

$150 USD in 2 days
(51 Reviews)
5.8
richestuser

I am talented PHP , mySQL ,javescript ,AJAX, WORDpress developer. I have very reasonable rate .I do WORDPRESS PHP , MySQL programming more than 5 years and within this period I had been working as slave coder.So, I More

$250 USD in 3 days
(60 Reviews)
5.6
wee493

A t2.micro should would for basic importing and everything . Would need to discuss how you'd be querying the data to get a more accurate guess on what EC2 server would be needed or if you'd need dedicated MySQL. I can More

$250 USD in 3 days
(87 Reviews)
5.9
pgu555d720e9981c

i will say i am perfect in doing these types of jobs i have also developed many websites with old csv data i will be fast and accurate in uploading full data to your server looking forward thanks

$55 USD in 3 days
(50 Reviews)
5.1
testotorko

Hi, I read carefully your project, and I wanna do it for you. If you wanna work together, send me a message to discuss details. I am an experienced Web Developer and Web Designer, and have worked with wide variety of s More

$150 USD in 11 days
(19 Reviews)
4.8
oda79

Hello! I am ready to help you with your task! Do you need an unattended script to do this import job? Please do not hesitate to contact me via messenger!

$250 USD in 20 days
(11 Reviews)
4.8
$155 USD in 3 days
(24 Reviews)
4.7
stoyanovalexandr

Hello, I am a skilled unix and network engineer with 5 years of experience in unix based systems, virtualization like proxmox,esxi,xen,virsh, iptv,zimbra etc. I like to automate system working with additional shell scr More

$111 USD in 1 day
(26 Reviews)
4.2
shiva1888

Hi I have been in this industry for 5 years and such jobs are my daily practice.I can assure you that if you work with me once, you will always work with me for these kind of projects.

$250 USD in 3 days
(4 Reviews)
3.4
$155 USD in 10 days
(4 Reviews)
3.1
djcarras

hi, i can help you with your project. According to the site, notifies the date of the last update, so I do not see that you need to run a process every hour, but I can do. send me more information about via freelancer More

$333 USD in 5 days
(2 Reviews)
2.4
stescovich

Hello, Thanks for reading my propusal. First of all let me tell you that loading such amount of data won't be fast. Since you need to do it in steps to prevent problems in DB. Secondly I would suggest you to r More

$100 USD in 20 days
(2 Reviews)
2.2
imran5757

Hi There, We can setup aws RDS instance which is highly available and the best performance database for mysql. Initially we would setup micro instance db if the queries are taking time to process then we can upgrade i More

$60 USD in 2 days
(4 Reviews)
2.1
mephri

Hello. Assuming data integrity is good (fields consistency and no junks) this could be done fast. Standard MySQL setup is sufficient (we are not talking about data access and processing here, right?) as long as the AWS More

$155 USD in 3 days
(0 Reviews)
0.0
arator

I can build this script using Python. I need only credentials of your database (IP, username, password, DB name) and may be an access to your AWS. Because I am not a client of AWS, I can suggest two possible cases of u More

$155 USD in 3 days
(0 Reviews)
0.0
gototakuya

I am an expert of full-stack web development and have similar experience with your requirement. although I am new to here freelancer and earn achievements for future so i will bid reasonable price. My planning is b More

$35 USD in 3 days
(0 Reviews)
0.0
kaippally

there could be easier ways to do this.

$111 USD in 5 days
(0 Reviews)
0.0
davideschirinzi

I would recommend to use: an rds with mysql or aurora as database with 20.000 provisioned IOps ebs volume (r3 instance family or I family depending on the needs of computing power of the queries). To import the file ca More

$250 USD in 3 days
(0 Reviews)
0.0
fb54ec414445702

How I understand, you need the application, which can automatically download the data from "http://download.companieshouse.gov.uk/en_output.html" every 60 minutes and makes import the data to SQL data base on MySQL. I More

$155 USD in 10 days
(0 Reviews)
0.0
eduardomurcia

I have the experience in work load of large files with management in record time. Also I have experience handling government databases with extensive confidentiality.

$111 USD in 2 days
(0 Reviews)
0.0