Java/Python development work on Red Sqirl

Public Clarification Board

• 
  Contest Holder

  • 5 years ago

  For clarity, the winning entry will be the first to put the package on the Red Sqirl market place (http://marketplace.redsqirl.com/) where we can check that it actually works.

  • 5 years ago
• 

  kenobrien76

  • 5 years ago

  Yes I get the fact that the docker image includes RedSqirl and an Hadoop Cloudera image. I'm a newbie to RedSqirl, but from looking at the docs it appears to be a data analysis tool based to the hadoop ecosystem. I didn't realize it was also responsible for adding data to HDFS. I just assumed you point it to your Hadoop and perform analysis on that. So my question on scaling 'around the data ingestion' how is this handled for large datasets, do I have a way to distribute this or is it just handled with threads in the generated package

  • 5 years ago
  1. 
     Contest Holder

     • 5 years ago

     Hi Kenneth, we will get back to you on your previous question. However, just wanted to let you know that the competition has been extended for a week. The task we would like to have completed it really straightforward once you familiarise yourself with Red Sqirl.
     
     1. Try to get Red Sqirl to run (e.g. using Docker).
     2. Try to understand how a Red Sqirl package works (explanation on the website www.redsqirl.com)
     All actions have an input and an output
     
     3. Try to create your own package (sample package on the website)
     4. Once you have the sample package running just change it to use the new REST data source.
     Get the REST data and use as the action output

     • 5 years ago
• 

  megaqujik

  • 5 years ago

  Hi,
  
  could you please give some hints what should be used in
  http://services.roadworks.org/sdep-publication/v5.2/sdep.asmx?op=RequestPull
  
  for
  Organisation fields, serviceUrl and time in DateTimeRequested="dateTime".

  • 5 years ago
  1. 
     Contest Holder

     • 5 years ago

     SOME ADVICE
     
     1. Try to get Red Sqirl to run (e.g. using Docker).
     2. Try to understand how a Red Sqirl package works (explanation on the website www.redsqirl.com)
     All actions have an input and an output
     
     3. Try to create your own package (sample package on the website)
     4. Once you have the sample package running just change it to use the new REST data source.
     Get the REST data and use as the action output

     • 5 years ago
• 

  rukkykf

  • 5 years ago

  is there any other way to get the image file. The docker pull request isn't going through. I've tried several times now, it's impossible for me to get the file because of my slow connection. However, if you have another way to distribute the image, perhaps as a downloadable tar file that I can download with a download manager (not docker), that would be nice.

  • 5 years ago
  1. 

     rukkykf

     • 5 years ago

     to be clearer: it will be easier for me to download the redsqirl image as a tar file and load it using docker load. You can create a tar file with docker save and host the tar file on a file sharing service like dropbox or google drive.

     • 5 years ago
  2. 
     Contest Holder

     • 5 years ago

     SOME ADVICE
     
     1. Try to get Red Sqirl to run (e.g. using Docker).
     2. Try to understand how a Red Sqirl package works (explanation on the website www.redsqirl.com)
     All actions have an input and an output
     
     3. Try to create your own package (sample package on the website)
     4. Once you have the sample package running just change it to use the new REST data source.
     Get the REST data and use as the action output

     • 5 years ago
• 

  karthiks416

  • 5 years ago

  Hi, Can you provide any other documentation. The provided link doesn't have much information like how to add REST and how to add data source

  • 5 years ago
  1. 
     Contest Holder

     • 5 years ago

     SOME ADVICE
     
     1. Try to get Red Sqirl to run (e.g. using Docker).
     2. Try to understand how a Red Sqirl package works (explanation on the website www.redsqirl.com)
     All actions have an input and an output
     
     3. Try to create your own package (sample package on the website)
     4. Once you have the sample package running just change it to use the new REST data source.
     Get the REST data and use as the action output

     • 5 years ago
• 
  Contest Holder

  • 5 years ago

  SOME ADVICE
  
  1. Try to get Red Sqirl to run (e.g. using Docker).
  2. Try to understand how a Red Sqirl package works (explanation on the website www.redsqirl.com)
  All actions have an input and an output
  
  3. Try to create your own package (sample package on the website)
  4. Once you have the sample package running just change it to use the new REST data source.
  Get the REST data and use as the action output

  • 5 years ago
• 
  Contest Holder

  • 5 years ago

  This is a relatively simple piece of work.

  • 5 years ago
• 
  Contest Holder

  • 5 years ago

  The competition has been extended for another week.

  • 5 years ago
• 
  Contest Holder

  • 5 years ago

  Thank you for trying.

  • 5 years ago
• 
  Contest Holder

  • 5 years ago

  Nobody managed to complete the task. We will review the details of this contest and may repost it with a simpler objective.

  • 5 years ago
• 

  kenobrien76

  • 5 years ago

  Hi. Have a couple of questions on this. What are the scaling requirements for this data ingestion application? Do you need live metrics as data is being added? Do you need an interface to add/update the data ingestion jobs? Where do you plan on executing the application (cloud, local server etc)? Thanks,

  • 5 years ago
  1. 
     Contest Holder

     • 5 years ago

     RedSqirl is inside the docker image. The docker is based on Cloudera image!
     you need create the package and run inside docker! The package should get the data using the Rest API and this should be the output of the action.
     Just follow the website instructions for create a package! Than you just need add the Rest call there.
     
     Here: https://redsqirl.com/pckdev.html

     • 5 years ago
• 

  kenobrien76

  • 5 years ago

  Also have you consider something like spring cloud data flow https://cloud.spring.io/spring-cloud-dataflow/ or is a custom built application a requirement?

  • 5 years ago
• 

  rukkykf

  • 5 years ago

  Do you have a sample open data source we can use for this contest that uses REST? The one you put in the contest description from the RoadWorks live API uses SOAP, not REST. I'm happy to proceed adding the data source from the RoadWorks API, but if you have another sample I could use instead, that'd be great.

  • 5 years ago
  1. 
     Contest Holder

     • 5 years ago

     You can try with any REST API on the website you can filter for API
     here is on example https://api.oireachtas.ie

     • 5 years ago
• 

  rukkykf

  • 5 years ago

  There are some other open data sources from the data.gov.uk site that use REST, should I proceed with one of those?

  • 5 years ago
  1. 
     Contest Holder

     • 5 years ago

     Yes, please go ahead with any other data source that you like.

     • 5 years ago
• 

  ITPyramid85

  • 5 years ago

  Hi, Sir. Can I implement your project on Offline Red Sqirl?

  • 5 years ago
  1. 
     Contest Holder

     • 5 years ago

     You can do it on Docker if you download it from the link provided.

     • 5 years ago
  2. 
     Contest Holder

     • 5 years ago

     Hi there, were you able to use Docker?

     • 5 years ago
• 
  Contest Holder

  • 5 years ago

  Hi there, there is some flexibility in that regard. What we want to assess is your ability to add a (data) package to Red Sqirl (based on a publicly available API). .
  
  For the API in question (i.e. https://data.gov.uk/dataset/1d9d6499-6ff5-4450-839a-68023c58452d/live-roadworks-api)
  
  We would be happy if you could bring in the location, start date/time, expected end date/time and description.

  • 5 years ago

Show more comments