The beginning of contributing to Telescope

Posted on by pluu3

Telescope is an open source project created to reignite development for Planet CDOT. Students (around 60 students) from Seneca College will be contributing for the development of this project. For more information and context please check the telescope overview.

Contributing:

Looking through the overview of telescope, we find a list of features that need to be implemented for the project to be created. The feature I wanted to implement was parsing the feed into a JSON file. I check the issues to find if the feature was already created, sadly it was and someone already wanted to work on it.

I decide not to give up on this feature, since I was really interested on the process of parsing XML to JSON. I commented on the issue that I wanted to aid the person who wanted on the feature too. To keep integrity, I wrote that I will create a commit/pull request but NOT let it get merge unless it gets approval from the person who originally wanted to work on the issue. The whole point of my code is to create a starting point for the project, since it is in it’s infancy. Hopefully I will be able to work together with the person who originally called on the issue.

Researching on parsing XML to JSON:

I start my search on trying to find methods/libraries on parsing the blog feeds(XML) to JSON. Looking through ways to convert feeds to JSON files, I ran into this blog. The blog the different method that can used to parse RSS feeds in JavaScript. I didn’t find the methods in the blog appealing so I continue my search. I found another article about how to convert XML to JSON, Converting Between XML and JSON. I also found that to be to much work, why reinvent the wheel when you don’t have too. I decide to look for a packages since the project was using NodeJS anyways.

Looking through packages:

The first package I will be looking through is rss-parser. Starting with the example code on the rss-parser page.

let Parser = require('rss-parser');
let parser = new Parser();
 
(async () => {
 
  let feed = await parser.parseURL('https://www.reddit.com/.rss');
  console.log(feed.title);
 
  feed.items.forEach(item => {
    console.log(item.title + ':' + item.link)
  });
 
})();

Going through this code, it seems to be exactly what we need, handles syncing, and also has some testing inside the package. Replacing the URL with my own blog feed URL, everything seemed fine in the beginning…

Problems with rss-parser:

I ran into the problem with package rss-parser in that I could not access the content_encoded tag. This tag display ALL of the blog post, the closes thing I could find in rss-parser was content which only showed the first few lines of the blog. I tested this with other students blog feeds but was still unable to get the full blog post in JSON, I could have missed something but after a long time Googling I did not find anything.

Looking through package rss-to-json I get exactly what I wanted. When the feed is loaded the JSON object also includes content_encoded tag.

var Feed = require('rss-to-json');
 
Feed.load('https://learnstartup.net/feed/', function(err, rss){
    console.log(rss);
});
 

Sadly the package doesn’t include testing and we also have to handle the syncing. I decide to upload a sample code with this package to create a starting point. Even if the code is riddled with bugs, it is OK since this gives more people the opportunity to work on something while the project is in it’s infancy.

Explaining my thought processes on GitHub:

I explain on GitHub that commit/pull request but NOT let it get merge unless it gets approval from the person who originally wanted to work on the issue.

The reason I am doing this is because I would like to see this project (with around 60 students working on it) to be implemented.

Purpose of my code:

  • If the original person working on the feature is stuck, maybe my code can provide a library/package that can help with the implementation person
  • Creating a starting point (Since little to no code has been implement at this point (Nov 11 2019))
  • If the code has problems/bugs, great. That means students/other people in the community can come in a work on bug fixes and issues creating plenty of work to do.

Addressing feedback and making changes:

I got some feedback from my pull request that I created. @humphd mentions that rss-to-json has no testing and may not be the right package to be used for this project, and I agree. @humphd provided a list of potential packages:

After some later testing I found that rss-to-json does not parse ATOM feeds, so I decide to scrape the idea/code.

Replying to feedback:

I reported back my findings, and explain the problems with rss-parser and rss-to-json. I created a table for more clarity:

PackagesNoteRecommend? ❌ ✔️
rss-parserI found the package rss-parser doesn’t show all the contents of the blog (Link to my blog for more information: https://wordpress.com/block-editor/post/paulopensourceblog.wordpress.com/1137) It would be great if someone was able to get it working but it didn’t for me @ImmutableBox❌
rss-to-jsonDoes not work with ATOM feeds @ImmutableBox❌
feedparserPull request created by @c3ho: #37🔨
htmlparser2Needs researching. (Feel free to contribute)🔨

The original person (@c3ho) who wanted to work on the issue also created a pull request for package feedparser which works with both feeds. Since I was given merging permissions, I was able to review @c3ho ‘s pull request.

Reviewing pull request:

Reviewing @c3ho ‘s pull request I found that the two files he created had the same code. This was probably a mistake so I address this issue in the review. I started testing the code to see if it does what it tend and found that it did, working with both feed inputs. I found that the JavaScript files committed did not have newlines at the end of them, so I also mentioned about that in the review. Overall the code was great, as long as the fixes are applied. Once the fixes are applied hopefully we will be able to merge the code onto the project.

Links:

Leave a comment

Published by pluu3