Java/Python development work on Red Sqirl
- Status: Closed
- Prize: $750
- Entries Received: 5
Contest Brief
We are looking for someone with Java/Python /Docker & REST skills to do the following:
1. Add open data sources to Red Sqirl platform (see www.redsqirl.com for more details) according to instructions which will be provided. See here for an introduction: https://redsqirl.com/pckdev.html
[ A list of open data sources will also be provided. A sample data source is: https://data.gov.uk/dataset/1d9d6499-6ff5-4450-839a-68023c58452d/live-roadworks-api ]
2. Build simple work flow to show that data source has been successfully added to Red Sqirl.
A Docker image on which you can develop and test your work can be found here: https://hub.docker.com/r/redsqirl/
[ If you have a Hadoop cluster you can also run Red Sqirl on that.]
Recommended Skills
Public Clarification Board
-
Contest Holder - 5 years ago
For clarity, the winning entry will be the first to put the package on the Red Sqirl market place (http://marketplace.redsqirl.com/) where we can check that it actually works.
- 5 years ago
-
kenobrien76
- 5 years ago
Yes I get the fact that the docker image includes RedSqirl and an Hadoop Cloudera image. I'm a newbie to RedSqirl, but from looking at the docs it appears to be a data analysis tool based to the hadoop ecosystem. I didn't realize it was also responsible for adding data to HDFS. I just assumed you point it to your Hadoop and perform analysis on that. So my question on scaling 'around the data ingestion' how is this handled for large datasets, do I have a way to distribute this or is it just handled with threads in the generated package
- 5 years ago
-
Contest Holder - 5 years ago
Hi Kenneth, we will get back to you on your previous question. However, just wanted to let you know that the competition has been extended for a week. The task we would like to have completed it really straightforward once you familiarise yourself with Red Sqirl.
1. Try to get Red Sqirl to run (e.g. using Docker).
2. Try to understand how a Red Sqirl package works (explanation on the website www.redsqirl.com)
All actions have an input and an output
3. Try to create your own package (sample package on the website)
4. Once you have the sample package running just change it to use the new REST data source.
Get the REST data and use as the action output- 5 years ago
-
megaqujik
- 5 years ago
Hi,
could you please give some hints what should be used in
http://services.roadworks.org/sdep-publication/v5.2/sdep.asmx?op=RequestPull
for
Organisation fields, serviceUrl and time in DateTimeRequested="dateTime".- 5 years ago
-
Contest Holder - 5 years ago
SOME ADVICE
1. Try to get Red Sqirl to run (e.g. using Docker).
2. Try to understand how a Red Sqirl package works (explanation on the website www.redsqirl.com)
All actions have an input and an output
3. Try to create your own package (sample package on the website)
4. Once you have the sample package running just change it to use the new REST data source.
Get the REST data and use as the action output- 5 years ago
-
rukkykf
- 5 years ago
is there any other way to get the image file. The docker pull request isn't going through. I've tried several times now, it's impossible for me to get the file because of my slow connection. However, if you have another way to distribute the image, perhaps as a downloadable tar file that I can download with a download manager (not docker), that would be nice.
- 5 years ago
-
rukkykf
- 5 years ago
to be clearer: it will be easier for me to download the redsqirl image as a tar file and load it using docker load. You can create a tar file with docker save and host the tar file on a file sharing service like dropbox or google drive.
- 5 years ago
-
Contest Holder - 5 years ago
SOME ADVICE
1. Try to get Red Sqirl to run (e.g. using Docker).
2. Try to understand how a Red Sqirl package works (explanation on the website www.redsqirl.com)
All actions have an input and an output
3. Try to create your own package (sample package on the website)
4. Once you have the sample package running just change it to use the new REST data source.
Get the REST data and use as the action output- 5 years ago
-
karthiks416
- 5 years ago
Hi, Can you provide any other documentation. The provided link doesn't have much information like how to add REST and how to add data source
- 5 years ago
-
Contest Holder - 5 years ago
SOME ADVICE
1. Try to get Red Sqirl to run (e.g. using Docker).
2. Try to understand how a Red Sqirl package works (explanation on the website www.redsqirl.com)
All actions have an input and an output
3. Try to create your own package (sample package on the website)
4. Once you have the sample package running just change it to use the new REST data source.
Get the REST data and use as the action output- 5 years ago
-
Contest Holder - 5 years ago
SOME ADVICE
1. Try to get Red Sqirl to run (e.g. using Docker).
2. Try to understand how a Red Sqirl package works (explanation on the website www.redsqirl.com)
All actions have an input and an output
3. Try to create your own package (sample package on the website)
4. Once you have the sample package running just change it to use the new REST data source.
Get the REST data and use as the action output- 5 years ago
-
Contest Holder - 5 years ago
This is a relatively simple piece of work.
- 5 years ago
-
Contest Holder - 5 years ago
The competition has been extended for another week.
- 5 years ago
-
Contest Holder - 5 years ago
Thank you for trying.
- 5 years ago
-
Contest Holder - 5 years ago
Nobody managed to complete the task. We will review the details of this contest and may repost it with a simpler objective.
- 5 years ago
-
kenobrien76
- 5 years ago
Hi. Have a couple of questions on this. What are the scaling requirements for this data ingestion application? Do you need live metrics as data is being added? Do you need an interface to add/update the data ingestion jobs? Where do you plan on executing the application (cloud, local server etc)? Thanks,
- 5 years ago
-
Contest Holder - 5 years ago
RedSqirl is inside the docker image. The docker is based on Cloudera image!
you need create the package and run inside docker! The package should get the data using the Rest API and this should be the output of the action.
Just follow the website instructions for create a package! Than you just need add the Rest call there.
Here: https://redsqirl.com/pckdev.html- 5 years ago
-
kenobrien76
- 5 years ago
Also have you consider something like spring cloud data flow https://cloud.spring.io/spring-cloud-dataflow/ or is a custom built application a requirement?
- 5 years ago
-
rukkykf
- 5 years ago
Do you have a sample open data source we can use for this contest that uses REST? The one you put in the contest description from the RoadWorks live API uses SOAP, not REST. I'm happy to proceed adding the data source from the RoadWorks API, but if you have another sample I could use instead, that'd be great.
- 5 years ago
-
Contest Holder - 5 years ago
You can try with any REST API on the website you can filter for API
here is on example https://api.oireachtas.ie- 5 years ago
-
rukkykf
- 5 years ago
There are some other open data sources from the data.gov.uk site that use REST, should I proceed with one of those?
- 5 years ago
-
Contest Holder - 5 years ago
Yes, please go ahead with any other data source that you like.
- 5 years ago
-
ITPyramid85
- 5 years ago
Hi, Sir. Can I implement your project on Offline Red Sqirl?
- 5 years ago
-
Contest Holder - 5 years ago
You can do it on Docker if you download it from the link provided.
- 5 years ago
-
Contest Holder - 5 years ago
Hi there, were you able to use Docker?
- 5 years ago
-
Contest Holder - 5 years ago
Hi there, there is some flexibility in that regard. What we want to assess is your ability to add a (data) package to Red Sqirl (based on a publicly available API). .
For the API in question (i.e. https://data.gov.uk/dataset/1d9d6499-6ff5-4450-839a-68023c58452d/live-roadworks-api)
We would be happy if you could bring in the location, start date/time, expected end date/time and description.- 5 years ago
How to get started with contests
-
Post Your Contest Quick and easy
-
Get Tons of Entries From around the world
-
Award the best entry Download the files - Easy!