Difference between revisions of "CSE503 SoftwareIdol"

From PublicWiki
Jump to: navigation, search
(Related Work)
 
(12 intermediate revisions by 2 users not shown)
Line 31: Line 31:
 
We're currently focusing '''popularity''', as measured by ratings and downloads.  As of 2/5, we're making a list of media-related software and running easy metrics against that.
 
We're currently focusing '''popularity''', as measured by ratings and downloads.  As of 2/5, we're making a list of media-related software and running easy metrics against that.
  
=== Possible Software ===
+
<font color="green">'''To-do list:
 +
* <strike>Cynthia is writing Dr. N. to try to schedule something for Friday, either after 4:30 (sigh) or between 10:30 and 12.</strike>
 +
* Alex is generating a text file with (up to) 100 pieces of media-related; OS; multiple-contributor software to start analyzing. This will include number of contributors and size/lines of code, per Ohloh.
 +
* Cynthia is collecting ratings/etc metrics for that software from various sites, and populating the table with that.
 +
* Suporn is figuring out what the right approach to counting search hits is, and populating the table with that.
 +
* Meet with Dr. N, 10:30am (ugh) Friday 2/8, 5th floor breakout.
 +
</font>
  
[Ohloh|http://www.ohloh.net] seems to be a good source.  An example of two pieces of media software:
+
=== OSS Software ===
  
http://www.ohloh.net/projects/3975?p=VLC
+
[http://www.ohloh.net Ohloh] seems to be a good source.  An example of two pieces of media software listings:
http://www.ohloh.net/projects/3137?p=guliverkli
 
  
== OSS Resources ==
+
*[http://www.ohloh.net/projects/3975?p=VLC VLC on Ohloh]
 +
*[http://www.ohloh.net/projects/3137?p=guliverkli guliverkli on Ohloh]
 +
 
 +
 
 +
'''Lists of Best Open-Source Software Lists:'''
 +
* http://www.opensourcewindows.org/
 +
* http://www.opensourcemac.org/
 +
* http://lifehacker.biz/articles/best-open-source-software/
 +
* http://www.opensourcelist.org/oss/suggestedapplications.html
 +
 
 +
The first software that was mentioned in all of the best open-source lists (some are listed below) is '''Firefox'''. So I started looking at Firefox and realize that the "add-ons" are open-source.  Though people can argue that anything we find in there might be Firefox-only characteristic, it's a good place to start looking with a smaller scope.
 +
 
 +
Firefox add-on browser has a feature to rank add-ons by popularity and ratings. This is a great example of ''when popularity and rating don't correlate''. For example, the [https://addons.mozilla.org/en-US/firefox/browse/type:1/cat:5/sort:rated top-rated download management] add-ons are far from the [https://addons.mozilla.org/en-US/firefox/browse/type:1/cat:5/sort:popular most popular ones]. The simple reasons behind this are:
 +
# there are only a few people who rates any one of the add-ons.
 +
# the ratings are not normalized by the number of people who rate the add-ons.
 +
 
 +
=== Ratings Sources ===
 +
 
 +
* [http://www.ohloh.net Ohloh] provides average rating, downloads, and [http://www.ohloh.net/about/faq#stack stacks]
 +
* [http://freshmeat.net Freshmeat] provides [http://freshmeat.net/stats/#rating rating], [http://freshmeat.net/stats/#vitality vitality], and [http://freshmeat.net/stats/#popularity popularity]
 +
* [http://fileforum.betanews.com Betanews] provides rating and downloads
 +
* [http://www.download.com/ Download.com] provides average editor ratings, user ratings, and downloads
 +
* [http://www.softsea.com/ Softsea] provides ratings
 +
* [http://www.snapfiles.com/ Snapfiles] provides editor ratings and user ratings
 +
 
 +
=== More on Metrics ===
 +
 
 +
'''Search hits:''' Experiments with using the same list to do web hit counts are interesting. Audacity, Gimp, Pidgin, Eraser, etc. are words with multiple meanings.  eMule, eMule morph and eMule Xtreme Mod are all separate downloads, but the hit counts overlap. Some things are too short a string (such as "ABC [Yet Another Bittorrent Client]").
 +
 
 +
== General OSS Resources ==
 
=== Cost of an OSS project ===
 
=== Cost of an OSS project ===
  
Line 57: Line 91:
 
=== Papers ===
 
=== Papers ===
  
Defining Open Source Software Project Success, Kevin Crowston, Hala Annabi, and James Howison, 2003 http://floss.syr.edu/publications/icis2003success.pdf
+
Defining Open Source Software Project Success, Kevin Crowston, Hala Annabi, and James Howison, 2003<br>
 +
http://floss.syr.edu/publications/icis2003success.pdf
 
This paper identify a range of measures that can be used to assess the success of open source software (OSS) projects.
 
This paper identify a range of measures that can be used to assess the success of open source software (OSS) projects.
  
Information Systems Success in Free and Open Source Software Development: Theory and Measures
+
Information Systems Success in Free and Open Source Software Development: Theory and Measures<br>
 
http://floss.syr.edu/publications/crowston2006flossSuccessSPIPpre-print.pdf
 
http://floss.syr.edu/publications/crowston2006flossSuccessSPIPpre-print.pdf
 +
 +
The perils and pitfalls of mining SourceForge<br>
 +
http://floss.syr.edu/publications/howison04msr.pdf
  
 
=== Useful Links ===
 
=== Useful Links ===
Line 76: Line 114:
 
A Case Study of Open Source Software Development: The Apache Server <br>
 
A Case Study of Open Source Software Development: The Apache Server <br>
 
http://conway.isri.cmu.edu/~jdh/collaboratory/research_papers/apachefinal3.pdf
 
http://conway.isri.cmu.edu/~jdh/collaboratory/research_papers/apachefinal3.pdf
 +
 +
How to Evaluate Open Source Software / Free Software (OSS/FS) Programs:<br>
 +
http://www.dwheeler.com/oss_fs_eval.html
 +
 +
 +
= Related Work =
 +
 +
FLOSS (Free/Libre and Open Source Software) includes both "free software" and "open source" which differs but share important characteristics. Researchers have been investigating FLOSS from different aspects.
 +
 +
Howison and Crowston reported the easily accessible sourceforge data to be difficult to gain meaningful result, because the data is dirty and needs careful screenning and deep analysis.
 +
 +
Crowston et al. investigates what success means within FLOSS context and examines the result by an empirical study using data from SourceForge.
 +
 +
Mockus et al. collected many claims/conventional wisdom about open source software and answered the claims by analyzing email archives of source code change history and problem reports.
 +
 +
David Wheeler wrote a manual for a user to pick an open source software.
 +
 +
Conceptual integrity is originally from Brooks, <u>The Mythical Man-Month</u>.
 +
 +
"Eyeballs" is actually Linus' Law, from Eric Raymond, <u>The Cathedral and the Bazaar</u>.

Latest revision as of 03:19, 12 March 2008

Project Ideas

We want to get involved in comparing OSS projects; there are a few different aspects of them that we could compare, and a few different metrics we could (try to) use for actually doing the comparison.

  • Aspects we could compare, from hard to easy:
    • Cost of building/maintaining
    • Successfulness
    • Popularity
    • Ratings
  • Metrics we could use:
    • Machine-learning of differentiating features (training a classifier)
    • "Englishyness"
    • Size (lines of code, number of revisions, etc)
    • Churn (rate of changes over time given repository change tracking)
    • Number of contributors
    • Other (many)

We probably want to combine a hard aspect with an easy metric or vice versa.

Options:

  • Come up with a reasonable way of estimating the "cost" of an OSS project; maybe do some metrics.
  • Come up with a reasonable way of defining "success"; analyze how a few not-too-deep metrics, such as size, or churn, or (whatever?) correspond to that split.
  • Take ratings or popularity, and measure some harder approaches (like Englishyness and churn) to try to come up with a correlation.
  • Take the easiest-to-measure aspects, get a good data set (ideally within a domain), and train a classifier using Weka or a similar toolkit; look at most-relevant features for something human-readable. http://en.wikipedia.org/wiki/Weka_(machine_learning)
    • A special case: Split a few code bases temporally and try to use classification of the repository for the *first year* of a project to relate to the success/cost/etc *eventually* (so as to come up with a predictor)

Also, this isn't a strict split, so: compare one or more aspect against one another - how does cost relate to ratings and popularity? How does popularity relate to success?

Current Work

We're currently focusing popularity, as measured by ratings and downloads. As of 2/5, we're making a list of media-related software and running easy metrics against that.

To-do list:

  • Cynthia is writing Dr. N. to try to schedule something for Friday, either after 4:30 (sigh) or between 10:30 and 12.
  • Alex is generating a text file with (up to) 100 pieces of media-related; OS; multiple-contributor software to start analyzing. This will include number of contributors and size/lines of code, per Ohloh.
  • Cynthia is collecting ratings/etc metrics for that software from various sites, and populating the table with that.
  • Suporn is figuring out what the right approach to counting search hits is, and populating the table with that.
  • Meet with Dr. N, 10:30am (ugh) Friday 2/8, 5th floor breakout.

OSS Software

Ohloh seems to be a good source. An example of two pieces of media software listings:


Lists of Best Open-Source Software Lists:

The first software that was mentioned in all of the best open-source lists (some are listed below) is Firefox. So I started looking at Firefox and realize that the "add-ons" are open-source. Though people can argue that anything we find in there might be Firefox-only characteristic, it's a good place to start looking with a smaller scope.

Firefox add-on browser has a feature to rank add-ons by popularity and ratings. This is a great example of when popularity and rating don't correlate. For example, the top-rated download management add-ons are far from the most popular ones. The simple reasons behind this are:

  1. there are only a few people who rates any one of the add-ons.
  2. the ratings are not normalized by the number of people who rate the add-ons.

Ratings Sources

More on Metrics

Search hits: Experiments with using the same list to do web hit counts are interesting. Audacity, Gimp, Pidgin, Eraser, etc. are words with multiple meanings. eMule, eMule morph and eMule Xtreme Mod are all separate downloads, but the hit counts overlap. Some things are too short a string (such as "ABC [Yet Another Bittorrent Client]").

General OSS Resources

Cost of an OSS project

A great advantage of Open-source software over a commercial software is the price. Users get the software for free, but there is some work that was put in to these great products. What are the actual costs of making these software?

Ohloh has an "estimated number of person-years" and associated estimated cost.

Quotes

"Indeed, as we have repeatedly emphasized, the Internet is the primary enabler of the OSS development and distribution process, making it possible for widely distributed groups to share ideas and software extremely quickly at negligible cost." Understanding Open Source Software Development By Joseph Feller, Brian Fitzgerald

"But open source is a low-cost way of increasing the opportunity for surprise." Lessons from Open Source software development, Tim O'Reilly 1999

Success of Open Source Project

Most downloaded on sourceforge: http://sourceforge.net/top/topalltime.php?type=downloads

Papers

Defining Open Source Software Project Success, Kevin Crowston, Hala Annabi, and James Howison, 2003
http://floss.syr.edu/publications/icis2003success.pdf This paper identify a range of measures that can be used to assess the success of open source software (OSS) projects.

Information Systems Success in Free and Open Source Software Development: Theory and Measures
http://floss.syr.edu/publications/crowston2006flossSuccessSPIPpre-print.pdf

The perils and pitfalls of mining SourceForge
http://floss.syr.edu/publications/howison04msr.pdf

Useful Links

Motivations of open-source developers:

Working for Free? Motivations for Participating in Open-Source Projects
http://mesharpe.metapress.com/media/fc22ht5ywp6vyndknrar/contributions/e/e/p/d/eepdf96rnt0geahv.pdf

Why Open Source software can succeed:
http://opensource.mit.edu/papers/rp-bonaccorsirossi.pdf

Case Studies: A Case Study of Open Source Software Development: The Apache Server
http://conway.isri.cmu.edu/~jdh/collaboratory/research_papers/apachefinal3.pdf

How to Evaluate Open Source Software / Free Software (OSS/FS) Programs:
http://www.dwheeler.com/oss_fs_eval.html


Related Work

FLOSS (Free/Libre and Open Source Software) includes both "free software" and "open source" which differs but share important characteristics. Researchers have been investigating FLOSS from different aspects.

Howison and Crowston reported the easily accessible sourceforge data to be difficult to gain meaningful result, because the data is dirty and needs careful screenning and deep analysis.

Crowston et al. investigates what success means within FLOSS context and examines the result by an empirical study using data from SourceForge.

Mockus et al. collected many claims/conventional wisdom about open source software and answered the claims by analyzing email archives of source code change history and problem reports.

David Wheeler wrote a manual for a user to pick an open source software.

Conceptual integrity is originally from Brooks, The Mythical Man-Month.

"Eyeballs" is actually Linus' Law, from Eric Raymond, The Cathedral and the Bazaar.