Auto News/Articles Importer to Wordpress
A system which crawls a news website every 15 minutes, scrape news titles, descriptions and images and import in wordpress directly. It makes sure that no duplicate records are imported. It also place news in proper categories and with proper tags and settings. Everything works in automated way without even single click. Technical stack used: CURL, LAMP, Custom PHP Framework, Wordpress
About Me
My main area of expertise includes Data Engineering (Small or Big Data level), Data Mining, ETL/ELT Data Pipelines, Data Automation and Automated Bots/Crawlers. I have experience of handling up to 100s of billions of records and 100s of TBs of structured and unstructured data; migration to Big Data/SQL data stores and handling real-time data streams. I have experience of large scale data collection/scraping from web, I developed countless crawlers in past and several includes data collection up to 100s of millions of records. Other areas of expertise includes Full Stack Data Science (Machine Learning/Deep learning). I also have a strong software engineering background and have experience in web development (MVC frameworks, Custom Development). I have skills to develop from scratch and customize exiting software/web applications. I have solution providing skills and can adjust myself to any technical stack, programming language or framework to provide the solution. Following is a brief list of technical stacks, programming languages and tools that I have expertise or used in past: Languages: Python (DJango, Flask), PHP (Laravel, Yii, Zend), Javascript (Angular, Typescript, Vue.js, Node.js), Java, C/C++, Perl, ASP, Matlab ------------------ Databases/Data Streaming: PostgreSQL, MySQL, MariaDB, SQL Server, SSIS, Oracle, NoSQL (Redis, HBase, Hive, MongoDB), Kinesis, Kafka, RabbitMQ, Solr ------------------ Big Data Ecosystem: Hadoop, HBase, Athena/Hive, Spark, Redshift ------------------ Cloud: AWS (EC2, ECS, RDS, S3, Redshift, Kinesis, ElastiCache/Redis, Lambda, Cloudwatch), GCP ------------------ Data Science/AI: Machine Learning (Classification, Clustering, Regression, SVM, K-NN, Naive Bayes, Random Forest, Linear Regression, Auto Regression), Deep Learning (Autoencoders, Stacked Autoencoders, CNN etc), Scikit-Learn, Scipy, Pandas, Data Wrangling, Web Scraping, Crawlers ------------------ Customization Experience: Opencart, Magento, Wordpress, Drupal, Zencart, other open source/custom made web applications ------------------ Other Skills: DevOps, CI/CD, CircleCI, Ansible, Docker, Deployer, Datadog, Openwhisk, Celery, Nutch, Thrift, Selenium, Jupyter Notebook, Git, REST Apis, SSH, TDD, Robotics, Good in adopting new stacks ------------------ Management: Agile, Scrum, Jira, Trello
