Building Customised Web Scrapping Tool

Project Objective: To build a database of LinkedIN users (and their career histories), who were involved with the training, implementation or consulting in relation to specified Business Software products during their careers. Inputs: Business Software Products (companies and/or branded product suites). Program functionality: The program will construct Product-Pairs from the Input list of products. The number of LinkedIn users who were involved with both products in each pair will be captured. Then, if possible, their career histories will also be captured in a relational table format. Output: - Excel sheet listing the Product-Pairs and the number of LinkedIn users. - Set of normalized tables (.txt or .csv) capturing the career histories of the users. Explanation: Let us say there are 10 software products A to J (each made by a different company) listed in the input database. A manual procedure would involve using the Advanced Search on LinkedIn, entering the names of the companies as Keywords (say, A & B, or in Current Employer: A and Past Employer: B), and then noting the number of results. Then we capture the career histories of all users which are listed in the above search (assuming the searcher has Premium membership access on LinkedIn). The desired program should automatically create all different combinations of companies A to J and execute the above steps. For a start we would test the program by using all ERP Software Products at [login to view URL] . Note: The list of Business Software Products will be disambiguated by searching for them on LinkedIn so as to remove duplicates, acronyms etc. The program should be simple enough that we can run it later on our own after specifying the input dataset – and on common platform such as Python or Java. Suggestions and comments from the programmer are welcome.