We have a requirement to do migration OLAP data warehouse system from on-Premise to Google Cloud -Big query Next year .
Data Size: 70 TB
Data: Structure data
Source DB: DB2/SQL/Oracle (Relation DB-OLAP) :ROW based DB
Target -Bigquery: No Relation DB , No SQL ,Columnar DB
Cloud Storage Layer: Google Bucket
CLoud Data warehouse : Bigquery
Data validation Requirment: How to validate file in google bucket vs data in Bigquery based on time stamp.
Use Case: Fraud Detection
Real time: Real time Initiala and detal load
Source: Bigquery - building Data Lake (No SQL)
Approach: Extract-> Load ->Transform
Technology: Data flow -Apache Beam using Java (Confirmed by client and Recommended by Google [login to view URL]
[login to view URL])
Language : Java (preferred) /Python
- How to migrate data from Source to target in real time ?
- How do data validate ,verification using Java/Python and generate report for data quality check like Row count, Cloumn count, Field value validation
- Special character, Non English text, sign ,blank value
- Data Modeling in Non Sql Big query Star schema to Data vault schema
- Data Masking and Data Encryption
- Data governance integration Informatica Axon and Colibra
- Data Migration Strategy
- Data Ingestion Strategy
- Date Quality and Data Verification Strategy
- List of Data Validation and verification approach and scenario
- List of data Transformation scenario
- Data Integration strategy
- Atunity VS Goldengate vs Talend vs Informatica vs IBM infostage vs MS BI SSIS vs Oracle ODI for Extract and Load comparison
12 freelancers are bidding on average ₹30324 for this job
hey there, I am a data science, I can help you in Need to do a migration of OLAP data warehouse system from on-Premise to Google Cloud -Big query Next year.
Hi Google Cloud Expert here, I have more then 2 years of experience in Google Cloud Platform and all its platform. It would be great working with the project Thanks Sneh Pandya