jump to navigation

Scrapy | A Fast and Powerful Scraping and Web Crawling Framework March 28, 2015

Posted by sandyclaus in Academic Technology, Programming, Python.
add a comment

An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

via Scrapy | A Fast and Powerful Scraping and Web Crawling Framework.

Web Scraping 101 with Python March 27, 2015

Posted by sandyclaus in Academic Technology, Programming, Python.
add a comment

Now that we have the packages we need, we can start scraping. But first, a couple of rules.

You should check a site’s terms and conditions before you scrape them. It’s their data and they likely have some rules to govern it.

Be nice – A computer will send web requests much quicker than a user can. Make sure you space out your requests a bit so that you don’t hammer the site’s server.

Scrapers break – Sites change their layout all the time. If that happens, be prepared to rewrite your code.

Web pages are inconsistent – There’s sometimes some manual clean up that has to happen even after you’ve gotten your data.

via Web Scraping 101 with Python.

Design of Computer Programs: Programming Principles – Udacity March 26, 2015

Posted by sandyclaus in Programming, Python.
add a comment

Course Summary

Understanding how to approach programming problems and devise a solution is an essential skill for any Python developer. In this course, you’ll learn new concepts, patterns, and methods that will expand your coding abilities from programming expert, Peter Norvig.

via Design of Computer Programs: Programming Principles – Udacity.

Python progression path – From apprentice to guru – Stack Overflow March 26, 2015

Posted by sandyclaus in Programming, Python.
add a comment

I’ve been learning, working, and playing with Python for a year and a half now. As a biologist slowly making the turn to bio-informatics, this language has been at the very core of all the major contributions I have made in the lab. I more or less fell in love with the way Python permits me to express beautiful solutions and also with the semantics of the language that allows such a natural flow from thoughts to workable code.

What I would like to know is your answer to a kind of question I have seldom seen in this or other forums. This question seems central to me for anyone on the path to Python improvement but who wonders what his next steps should be.

via Python progression path – From apprentice to guru – Stack Overflow.