Data Engineer Jobs

I am in dire need of guidance to efficiently set up Meltano for data pipelining primarily from relational databases such as MySQL and PostgreSQL. The goal is not only to have a system in place but also to grasp a deep understanding of the whole process. Key responsibilities: - Set up Meltano for data pipelining from relational databases - Offer expertise and guidance throughout the setup process Ideal Candidate: - Expertise with Meltano and data pipelining - Experience with relational databases particularly PostgreSQL - Excellent communication and teaching skills - Attention to detail, patience and willingness to explain complex concepts in a digestible manner. I look forward to collaborating with a professional who can offer high quality assistance, and expedite the learning and imple...

I'm seeking a knowledgeable specialist to perform benchmarks on my Pgvector Chroma Faiss database. The key objectives for this project are: - Analyzing speed and query performance. The ideal freelancer should have: - Proficiency with Pgvector, Chroma, and Faiss. - Experience in database benchmarking, with a focus on query performance metrics. - Ability to provide detailed reports on performance metrics. - Ability to offer performance tuning recommendations based on benchmark results. As this is for personal use, the project requires a specialist who can deliver thorough insights without the need for a business-oriented approach. Your technical database expertise will be crucial for this project.

I'm urgently in need of a qualified developer or data engineer to create a pipeline utilizing DBT and SQLITE. The goal is to process a multitude of json files housed in a particular folder. Please refer to the assignment document for detailed instructions Step 1. Initial Setup: ● Use DBT and SQLite to create a pipeline that processes files from a folder structure. You can access the folder here Step2: Adaptation for Continuous File Deliveries: ● Adjust the pipeline to handle continuous file deliveries, where new folders with JSON files are added over time. For example: Step 3: Data Quality Assurance: ● Create a suite of data checks to ensure the data is clean and add a DBT command to run these checks. ● Suggest additional information or metrics that could be used to further ensure ...